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INTRODUCTION 


The  purpose  of  this  report  is  to  present  a plan  for 
evaluating  the  impact  of  the  Air  Force  Clinical  Laboratory 
Automation  System  (AFCLAS)  on  the  operation  and  management 
of  the  clinical  laboratory  and  on  users  and  beneficiaries 
of  laboratory  results  outside  the  clinical  laboratory. 

This  plan  is  called  an  evaluation  plan.  The  results  of  the 
evaluation  are  intended  to  aid  those  who  must  decide  whether 
the  AFCLAS  system  should  be  terminated,  continued  where 
already  installed,  or  introduced  at  other  Air  Force  medical 
centers.  The  results  should  also  provide  a basis  for 
recommending  changes  that  could  enhance  the  operation  of 
AFCLAS.  The  plan  calls  for  evaluation  of  changes  in  the 
laboratory  that  result  from  the  introduction  of  AFCLAS. 

Thus,  two  periods  of  data  collection  are  planned;  Period  X, 
prior  to  installation  of  AFCLAS,  and  Period  Y,  after  AFCLAS 
is  operational.  Minor  modifications  to  the  data  collection 
procedures  will  have  to  be  made  before  Period  Y,  based  on 
the  actual  operation  of  AFCLAS. 


AFCLAS  is  an  automated  data  processing  and  management 
system  that  the  Air  Force  is  installing  in  the  clinical 
laboratories  at  two  test  sites,  both  of  which  are  Air  Force 
hospitals : 

• USAF  Medical  Center,  Wr ight-Patter son  AFB,  Ohio 

• Malcolm  Grow  USAF  Medical  Center  at  Andrews  AFB, 
Washington,  D.C. 


The  immediate  goal  of  AFCLAS  is  to  improve  the  operation 
and  management  of  the  clinical  laboratory  and  thus  ultimately 
to  enhance  the  laboratory's  contribution  to  quality  patient 
care  and  to  the  patient's  health  status.  It  is  anticipated 
that  the  following  improvements  will  result  from  the  intro- 
duction of  AFCLAS: 


'N... 


'raCEDIMi  P.CE  PIW.:d 


1 


• Shorter  laboratory  report  turnaround  time 

• Increased  laboratory  staff  productivity 

• Improved  accuracy,  format,  and  content  in  reports 
of  laboratory  results 

• Improved  capability  for  evaluating  laboratory 
performance 

• Easier  and  more  rapid  access  to  laboratory  records. 
AFCLAS  provides  for  accomplishing  these  objectives  by: 

• Maintaining  for  the  individual  patient  internal  data 
files  that  are  rapidly  accessible  for  data  entry 
and  retrieval 

• Providing  online  monitoring  of  continuous  flow 
laboratory  instruments 

• Automating  clerical  tasks  such  as  preparation  of 
worksheets  and  specimen  collection  lists 

• Screening  all  test  results  and  flagging  those  out- 
side normal  limits  and  those  exceeding  specified 
technical  limits  (e.g.,  incompatible  with  life) 

• Providing  daily  reports  and  cumulative  patient 
reports  at  regular  intervals  and  current  test 
result  information  on  demand  at  I/O  devices 

• Automatically  printing  results  for  stat  (emergency) 
orders  as  soon  as  they  are  entered  and  verified 

• Providing  quality  control  tools  such  as  statistical 

analyses  and  monitoring  of  the  results  of  tests  on\ 
quality  control  specimens.  \ 

While  AFCLAS  will  have  little  or  no  effect  on  the  tech- 
nical methods  and  procedures  used  by  technicians  in  conducting 
the  laboratory  determinations,  the  improvements  listed  should 
contribute  to  improved  quality  of  the  analytical  output.  In 
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addition  to  providing  better  tools  for  quality  assessment, 
AFCLAS  should  make  more  of  the  technician's  time  available 
for  technical  (as  opposed  to  clerical)  tasks.  The  reduction 
in  clerical  tasks  should  also  make  laboratory  results  less 
vulnerable  to  human  error  in  computation,  recording,  and 
transcription  because  more  time  is  available  to  conduct  the 
tests  and  check  results. 

Another  important  objective  of  AFCLAS  is  that  its 
operation  and  effects  be  acceptable  to  the  persons  upon  whom 
they  impinge.  In  the  case  of  laboratory  staff,  acceptance  is 
a prerequisite  to  any  of  the  improvements  listed.  Outside 
the  laboratory,  personnel  affected  are  physicians  and  staffs 
of  outpatient  clinics,  nursing  units,  and  the  medical  records 
department.  In  these  areas,  the  impact  of  AFCLAS  will  be 
manifested  through  the  laboratory  reports.  New  laboratory 
report  forms  should  change  the  time  required  for  filing  the 
reports  in  outpatient  records  and  in  inpatient  charts.  Most 
important,  the  accuracy,  usability,  and  timeliness  of  reports 
sent  to  physicians  should  be  affected.  Patients  will  be 
affected  through  the  service  they  receive  from  the  laboratory. 
This  includes  waiting  time,  the  attitudes  of  laboratory  staff 
members,  and  availability  of  their  laboratory  results  when 
needed  by  a physician. 

The  technical  aspects  of  AFCLAS  are  defined  in  the  RFP 
performance  specifications  and  in  the  subsequent  contract 
with  Honeywell,  Inc.  These  technical  aspects  of  AFCLAS  will 
have  been  evaluated  prior  to  acceptance  of  the  system  by  the 
Air  Force.  Therefore,  it  is  assumed  that  when  data  are  being 
collected  under  this  plan  following  introduction  of  AFCLAS, 
AFCLAS  will  be  operating  in  complete  compliance  with  the  RFP 
specifications  and  with  the  terms  of  the  contract. 
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This  evaluation  plan  provides  for  evaluating  AFCLAS  by: 

(1)  investigating  statistically  some  of  the  changes  that  are 
hypothesized  effects  of  AFCLAS;  (2)  studying  changes  in 
personnel  time  required  to  perform  functions  that  AFCLAS 
supports;  (3)  estimating  dollar  benefits  and  costs  of  intro- 
ducing AFCLAS;  and  (4)  surveying  the  acceptance  of  and  satis- 
faction with  AFCLAS  by  personnel  inside  and  outside  the 
laboratory  and  by  patients.  Effects  of  AFCLAS  to  be  investi- 
gated in  special  detail  include  changes  in  completeness  of 
the  medical  record,  timeliness  of  laboratory  reports,  tele- 
phone calls  to  the  laboratory,  errors  in  arriving  outpatient 
request  slips,  and  service  time  at  the  reception  desk. 

The  dollar  benefits  and  costs  analysis  will  estimate  the 
net  dollar  cost  of  introducing  AFCLAS,  including  costs 
associated  with  changes  effected  by  AFCLAS  inside  and  outside 
the  laboratory  and  the  cost  of  AFCLAS  itself.  Since  not  all 
of  the  changes  anticipated  can  be  quantified  in  dollars,  a 
positive  net  cost,  if  found,  must  be  considered  as  the  price 
of  benefits  that  are  not  quantified  in  dollars.  The  dollar 
benefits  and  costs  analysis  will  be  based  on  data  collected 
at  each  test  site  before  and  after  installation  of  AFCLAS. 

The  acceptance  and  satisfaction  analysis  will  compare 
the  prior  receptiveness  and  actual  response  to  AFCLAS  of  the 
laboratory  staff,  physicians,  hospital  support  staff,  and 
patients.  The  measures  of  acceptance  and  satisfaction  will  be 
scores  derived  from  responses  to  questionnaires.  Acceptance 
and  satisfaction  might  be  expected  to  affect  such  factors  as 
personnel  performance  and  personnel  turnover.  These  factors 
in  turn  will  affect  dollar  benefits  and  costs.  Personnel 
performance  will  be  reflected  to  a limited  extent  by  the  time 
to  perform  clerical  tasks  and  tasks  related  to  AFCLAS.  How- 
ever, no  attempt  will  be  made  to  measure  the  effect  of  AFCLAS 


on  turnover  because  the  period  planned  for  the  evaluation  is 
too  short  to  permit  significant  changes  to  occur. 

Evaluation  of  AFCLAS  will  require  that  data  on  laboratory 
operations  be  collected  before  and  after  AFCLAS  is  installed 
and  in  such  a manner  as  to  minimize  interference  with  those 
and  other  facility  operations.  Therefore,  wherever  possible, 
data  routinely  generated  will  be  used. 

The  evaluation  plan  is  designed  to  reflect  actual  changes 
in  the  laboratory  as  accurately  as  possible  subject  to  the 
limitations  imposed  by  the  situation  in  which  the  data  must 
be  collected.  Some  of  the  factors  that  could  cause  the  data 
collected  to  give  an  inaccurate  representation  of  the  true 
laboratory  situation  are: 

• Seasonal  variation  in  number  and  distribution  of 
laboratory  requests 

• Short-term  variation  in  nuniber  and  distribution  of 
laboratory  requests 

• Effects  of  administrative  operation  and  technical 
changes 

• Effects  of  outside  observers  on  laboratory  and 
hospital  operations 

• Limited  manpower  available  for  data  collection. 

These  problems  were  considered  when  designing  the  study 
and  the  evaluation  plan  attempts  to  minimize  their  effect 
through  a careful  study  design  and  normalization  of  results 
for  changes  not  introduced  by  AFCLAS. 

Section  II  discusses  the  hypotheses  to  be  investigated 
and  Section  III  describes,  in  general  terms,  the  data  to  be 
collected  and  several  assumptions  fundamental  to  the  evalua- 
ation.  An  overview  of  the  analysis  is  presented  in  Section  IV. 
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Section  V outlines  an  approach  to  estimating  the  effects  of 
introducing  AFCLAS  in  facilities  other  than  the  test  sites. 

A brief  summary  of  the  main  sections  of  the  evaluation  plan 
is  given  in  Section  VI.  Appendix  A contains  the  definitions 
of  terms  used  in  the  plan.  Appendix  B is  a data  collection 
handbook  describing  the  details  of  data  collection.  A dis- 
cussion of  the  survey  schedules  (or  questionnaires)  and 
copies  of  the  Period  Y version  of  the  questionnaires  are  con- 
tained in  Appendix  C.  In  December  1974  a pretest  of  the 
evaluation  plan  was  conducted  at  an  Air  Force  hospital  similar 
in  size  and  services  to  the  two  test  sites.  The  procedures 
used  for  that  pretest  are  described  in  a 12  December  1974 
paper  entitled  Pretest  for  Evaluation  Plan  for  the  Air  Force 
Clinical  Laboratory  Automation  System  (AFCLAS) . This  paper 
is  included  as  Appendix  D to  this  evaluation  plan. 


II.  HYPOTHESES  TO  BE  INVESTIGATED 


Study  of  the  objectives  and  operational  specifications 
for  AFCLAS,  review  of  relevant  literature,  and  observation  of 
the  manual  information  systems  in  the  clinical  laboratories 
at  Malcolm  Grow  Medical  Center  and  Wr ight-Patterson  Medical 
Center  led  to  identification  of  various  planned  and  potential 
impacts  of  introducing  AFCLAS  in  place  of  the  existing  infor- 
mation system.  The  anticipated  importance  of  each  impact, 
the  possibility  of  measuring  it,  and  the  feasibility  of 
collecting  data  to  study  the  impact  were  considered.  As  a 
result,  58  hypotheses  were  defined  for  use  in  the  evaluation. 
These  hypotheses  concern  the  effects  of  AFCLAS,  including 
effects  specifically  planned  in  the  Request  for  Proposals  (RFP) 
on  users  inside  and  outside  the  laboratory  and  on  patients. 

The  term  "hypotheses"  used  here  does  not  imply  that  statistical 
hypothesis  testing  will  be  applied  in  each  case.  However,  when 
ever  appropriate,  validity  of  the  hypotheses  will  be  tested 
using  standard  statistical  techniques.  The  hypotheses  are  in- 
tended primarily  to  serve  as  a guide  for  data  collection  and 
analysis . 

The  data  required  to  study  the  hypothesized  effects  will 
be  explained  in  more  detail  in  Section  III  and  in  Appendix  B 
on  data  collection.  Data  are  needed  from  each  test  site  for 
two  data  collection  periods  (a  total  of  four  data  collection 
efforts),  one  before  AFCLAS  is  installed  (Period  X)  and  one 
after  AFCLAS  is  in  operation  (Period  Y) . The  two  sets  of 
data  from  each  test  site  will  be  used  to  identify  and  quantify 
changes  that  occur  between  the  two  data  collection  periods. 
Generally,  data  to  be  collected  must  be  identified,  and  all 
preparations  for  collection  must  be  accomplished  in  advance 
of  Period  X.  Data  to  be  collected  on  new  tasks  generated 
by  AFCLAS  are  an  exception,  since  no  data  concerning  them 


7 


will  exist  during  Period  X.  These  tasks  can  be  identified 
and  hypotheses  concerning  their  effects  can  be  incorporated 
in  the  evaluation  plan  after  AFCLAS  is  installed  but  before 
Period  Y begins.  To  preserve  the  cbjectivity  and  validity 
of  the  study,  all  hypotheses  should  be  included  in  the 
analysis,  and  no  new  hypotheses  should  be  added  after  the 
start  of  Period  Y data  collection. 

* i"  i 

The  hypotheses  to  be  investigated  are:  ~ 

1.  Time  spent  producing  administrative  reports  will 
change,  for  an  equal  number  and  .xStribution  of 
test  requests. 

2.  The  number  of  telephone  inquiries  to  the  laboratory 
central  site  (the  reception  desk  or  central  computer 
site)  will  change,  for  an  equal  number  and  distri- 
bution of  test  requests  and  an  equal  number  of  tele- 
phone lines  and  personnel  at  the  central  site. 

3.  Time  spent  responding  to  telephone  inquiries  received 
by  the  laboratory  central  site  will  change,  for  an 
equal  number  and  distribution  of  test  requests  and 

an  equal  number  of  telephone  lines  and  personnel  at 
the  central  site. 

4.  Time  spent  on  the  telephone  for  inquiries  to  the 
laboratory  by  personnel  both  inside  and  outside  the 


* See  Appendix  A for  definitions  of  terms. 

t Accepting  an  hypothesis  as  stated  above  is  equivalent 
to  rejecting  the  null  hypothesis  if  the  hypothesis  was 
worded  in  standard  statistical  terminology. 

j:  There  is  a total  of  57  hypotheses.  The  list  contains  58 

items  because  hypothesis  #36  was  dropped  prior  to  the 
start  of  Period  X data  collection. 
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laboratory  will  change,  for  an  equal  number  and 
distribution  of  test  requests  and  an  equal  number  of 
telephone  lines  and  personnel  at  the  central  site. 

5.  The  distribution  by  location  and  type  of  caller  of 
telephone  inquiries  to  the  laboratory  central  site 
will  change  for  an  equal  number  and  distribution  of 
test  requests  and  for  an  equal  number  of  telephone 
lines  and  personnel  at  the  central  site. 

6.  Time  required  for  filing  laboratory  clinical  forms 
in  laboratory  files  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

7.  Time  required  to  file  laboratory  reports  in  out- 
patient medical  records  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 

8.  Time  required  to  file  laboratory  reports  at  the 
nursing  stations  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

9.  Time  required  for  preparing  laboratory  clinical 
forms  will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

; 10.  Time  spent  compiling  College  of  American  Pathologists 

( (CAP)  standard  workload  figures  will  change,  for  an 

• equal  number  and  distribution  of  test  requests. 

11.  Time  required  for  labeling  specimens  will  change, 

for  an  equal  number  and  distribution  of  test  requests. 

ji  12.  Time  required  for  preparing  lists  for  specimen 

collection  will  change,  for  an  equal  number  and 
distribution  of  test  requests. 

13.  Time  required  for  recording  results  of  tests  that 

will  be  online  with  APCLAS  will  change,  for  an  equal 
number  and  distribution  of  test  requests 
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14.  Time  required  for  recording  results  of  tests  that 
will  be  offline  with  AFCLAS  will  change,  for  an 
equal  ntamber  and  distribution  of  test  requests. 

15.  Time  required  for  entering  laboratory  requests  by 
mark  sense  card  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

16.  Time  required  for  keyboard  entry  of  patient  name  and 
identification  at  the  laboratory  reception  desk  will 
change,  for  an  equal  number  and  distribution  of  test 
requests . 

17.  Time  required  for  keyboard  entry  of  patient  name 
and  identification  in  the  registrar's  office  will 
change,  for  an  equal  number  and  distribution  of  test 
requests . 

18.  Time  required  for  entering  laboratory  test  results 
by  mark  sense  card  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

19.  Time  required  for  keyboard  entry  of  free-text 
laboratory  test  results  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 

20.  Time  required  for  supervisor's  review  and  certifi- 
cation of  results  and  worksheets  will  change,  for 
an  equal  number  and  distribution  of  test  requests. 

21.  Time  required  for  technician's  review  and  certification 
of  results  and  worksheets  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 

22.  Time  spent  on  statistical  analysis  for  quality  control 
will  change,  for  an  equal  number  and  distribution  of 
test  requests. 
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23.  The  number  of  times  statistical  analyses  of  quality 
control  sample  results  are  performed  will  change,  for 
an  equal  number  and  distribution  of  test  requests. 

24.  The  kinds  of  statistical  analyses  of  quality  control 
sample  results  will  change,  for  an  equal  number  and 
distribution  of  test  requests. 

25.  The  number  and  kinds  of  statistical  analyses  of 
patient  results  by  population  will  change,  for  an 
equal  number  and  distribution  of  test  requests. 

26.  The  time  required  for  statistical  analyses  of  patient 
results  by  population  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

27.  Time  spent  on  calculations  and/or  conversions  from 
raw  to  clinical  values  for  test  reports  will  change, 
for  an  equal  number  and  distribution  of  test  requests. 

28.  Time  spent  on  operation  of  the  computer  by  the  labora- 

★ 

tory  staff  will  change. 

29.  Kinds  and  numbers  of  laboratory  clinical  forms  used 
will  change,  for  an  equal  number  and  distribution  of 
test  requests. 

30.  Kinds  and  numbers  of  tests  sent  out  to  other  labora- 
tories for  processing  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

31.  Kinds  and  numbers  of  tests  sent  to  the  laboratory 
being  studied  by  other  laboratories  will  change. 


* Time  spent  by  laboratory  personnel  in  computer  operation 
does  not  include  functions  performed  at  remote  terminals 
or  input  devices  which  are  covered  by  other  hypotheses; 
additional  personnel  added  to  the  laboratory  staff  specifi- 
cally to  operate  the  AFCLAS  system  will  be  included  as  a 
cost  of  introducing  AFCLAS. 
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for  an  equal  number  and  distribution  of  test  requests 
at  the  laboratories  sending  the  test  requests. 

32.  The  number  of  atat  requests  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 

33.  Time  that  laboratory  staff  members  work  in  addition 
to  normal  duty  hours  will  change,  for  an  equal  number 
and  distribution  of  test  requests,  and  an  equal  number 
of  laboratory  staff. 

34.  Usability  of  laboratory  reports  to  physicians  will 
change. 

35.  Time  to  provide  patient  service  at  the  laboratory 
reception  desk  will  change,  for  an  equal  number  and 
distribution  of  test  requests. 

36.  (This  hypothesis  was  dropped  prior  to  Period  X.) 

37.  Time  the  patient  spends  at  the  laboratory  will  change, 
for  an  equal  number  and  distribution  of  test  requests. 

38.  Patient  satisfaction  with  the  laboratory  will  change. 

39.  Physician  acceptance  of  the  laboratory  will  change. 

40.  Laboratory  staff  satisfaction  will  change. 

41.  Admissions  and  Dispositions  Department  staff 
acceptance  of  the  laboratory  will  change. 

42.  Medical  Records  Department  staff  acceptance  of  the 
laboratory  will  change. 

43.  Registered  nurses'  acceptance  of  the  laboratory  will 
change. 

44.  The  number  of  transcription  errors  will  change,  for 
an  equal  number  and  distribution  of  test  requests. 
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45.  Completeness  of  the  medical  record  will  change. 

46.  Turnaround  time  will  change,  for  an  equal  number  and 
distribution  of  test  requests. 

47.  Quality  of  patient  care  will  change. 

48.  Numbers  and  kinds  of  reports  produced  will  change. 

49.  Numbers  and  kinds  of  tests  requested  by  physicians 
for  outpatients  will  change,  for  an  equal  number  of 
outpatient  visits,  for  the  same  distribution  of 
utilization  by  outpatient  departments. 

50.  Numbers  and  kinds  of  tests  requested  by  physicians 
for  inpatients  will  change,  for  an  equal  number  of 
total  hospital  inpatient  days,  for  the  same  distri- 
bution of  patients  by  inpatient  category. 

51.  Time  required  to  complete  laboratory  test  request 
slips  outside  the  laboratory  will  change,  for  an 
equal  number  and  distribution  of  test  requests. 

52.  The  numbers  of  improperly  completed  outpatient  test 
request  slips  arriving  at  the  reception  desk  will 
change,  for  an  equal  number  and  distribution  of  test 
requests . 

53.  Time  required  by  supervisors  (laboratory  officers) 
to  respond  to  inquiries,  complaints,  errors  detected 
before  test  results  are  reported,  and  errors  in  test 
results  reported  to  physicians  will  change,  for  an 
equal  number  and  distribution  of  test  requests. 

54.  Time  between  distribution  and  filing  of  laboratory 
reports  will  change. 

55.  Time  required  for  completing  cumulative  laboratory 
reports  (equivalent  to  flow  sheets  of  laboratory 
results)  will  change. 
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56.  Time  required  to  report  stat  results  by  telephone 
will  change,  for  an  equal  number  and  distribution 
of  test  requests. 

57.  Time  required  for  preparing  a list  for  the  laboratory 
of  admissions,  interward  transfers,  and  discharges 
will  change,  for  an  equal  number  and  distribution  of 
test  requests. 

58.  Time  required  for  reviewing  new  AFCLAS-generated 
reports  will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

A number  of  potential  effects  of  AFCLAS  that  have  been 
identified  are  not  included  in  the  above  hypotheses.  Thus, 
they  are  not  specifically  addressed  by  the  data  collection 
and  analyses.  They  are  excluded  for  one  or  more  of  the 
following  reasons: 

• The  estimated  magnitude  of  the  change  was  small. 

• The  change  could  not  be  defined  precisely. 

• The  data  collection  would  unduly  disrupt  laboratory 
or  hospital  operation. 

• Data  collection  would  be  too  expensive  for  the 
information  gained. 

The  potential  changes  not  specifically  included  are  listed 
below:  <■  ' 

1.  The  time  laboratory  personnel  spend  walking 

2.  Time  spent  by  technicians  to  begin  online  data 
collection  each  morning  and  after  each  interruption 
of  computer  operation 

3.  The  time  lost  with  AFCLAS  in  correcting  improperly 
completed  mark- sense  request  forms  and  improperly 
marked  test  results  and  time  lost  in  reentering  the 
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corrected  cards  (If  preliminary  observation,  after 
AFCLAS  is  installed  but  before  Period  Y,  shows  that 
this  time  is  significant,  the  hypothesis  can  be 
added  and  listed.) 

4.  Time  spent  in  maintaining  backup  system  proficiency 

5.  A change  in  the  laboratory  staff's  average  skill 
level 

6.  A change  in  the  number  of  tests  that  are  repeated 
because  of  errors  in  the  laboratory 

7.  A change  in  the  number  of  physician  requests  for 
repeat  tests 

8.  A change  in  accuracy  of  test  performance  (as  con- 
trasted with  accuracy  in  transcription  of  results) 

9.  A decrease  in  the  number  of  reports  going  to  physicians 
with  results  that  are  inconsistent  with  patient  life 

10.  Changes  in  time  spent  in  quality  control  other  than 
time  spent  on  calculations 

11 o A change  in  the  number  of  improperly  completed 
inpatient  request  slips 

12.  A change  in  the  number  of  physician  walk-in  inquiries 
to  the  laboratory 

13o  A change  in  the  utility  of  administrative  and  quality 
control  reports 

14.  A change  in  the  number  and  kind  of  quality  control 
reports 

15.  A change  in  the  technician  time  involved  in  returning 
the  laboratory  to  computer  support  when  AFCLAS  goes 
down  (Actual  time  spent  operating  the  AFCLAS  computer 
is  included  in  an  hypothesis.) 


A 
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III.  DATA  COLLECTION  AND  ASSUMPTIONS 


This  section  describes  in  general  terms  the  data  collection 
effort  required  to  implement  the  evaluation  plan  as  well  as 
several  assumptions  especially  fundamental  to  the  evaluation. 

The  specific  data  to  be  collected,  methods  for  estimating 
sample  sizes,  and  methods  for  collecting  the  data  are  con- 
tained in  Appendix  B,  Data  Collection  Handbook. 

The  evaluation  plan  requires  that  data  be  collected  before 
AFCLAS  is  installed  and  again  after  it  is  fully  opera-' 'onal  to 
investigate  the  hypotheses  listed  in  Section  II.  Ideally, 

Period  Y,  the  data  collection  period  after  AFCLAS  is  imple- 
mented, should  not  begin  until  well  after  AFCLAS  becomes 
operational  and  is  functioning  properly  to  support  the  labora- 
tories (considering  the  risk  of  intervening  factors*)  because 
of  slow  user  acceptance.  This  phenomenon  of  resistance  to 
innovation  or  change,  called  aultuval  lag,  is  well  documented 
in  the  sociological  literature.  This  suggests  that  acceptance, 
satisfaction,  and  enthusiasm  for  AFCLAS  would  increase  for 
at  least  several  years  after  the  system  becomes  operational. 

The  Battelle  study  of  the  Technicon  Medical  Information 
System  at  El  Camino  Hospital  in  California  found  that  physician 
acceptance  of  the  system  continued  to  increase  for  over  2 years 
after  the  system  was  operational.  Assuming  that  AFCLAS  becomes 
operational  in  3 to  5 months  after  the  end  of  Period  X,  then 
the  data  collection  for  Period  Y might  start  6 months  after 
operation,  which  would  be  1 calendar  year  following  the  begin- 
ning of  Period  X,  in  order  to  minimize  seasonal  variations. 
However,  it  has  been  decided  by  the  Directorate  of  Medical 
Plans  and  Resources,  Office  of  the  Surgeon  General,  Headquarters 
United  States  Air  Force,  that  Period  Y will  begin  3 months 


* See  Appendix  A. 
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after  acceptance  of  AFCLAS„  Three  months  is  the  minimum  time 
acceptable  to  allow  the  laboratory  to  make  the  transition  to 
smooth  automated  operation.  The  periods  of  data  collection 
will  be  4 weeks  for  most  data  items  and  several  months  for  a 
few  data  items.  The  length  of  the  actual  data  collection 
period  is  determined  by  the  required  sample  size  and  physical 
limitations  of  sampling. 

The  time  specified  for  Period  X data  collection  should 
be  selected  after  considering  the  following  factors,  which 
will  influence  data  collection  or  analysis: 

• Any  changes  in  laboratory  operations  implemented 
before  or  during  Period  X in  anticipation  of  AFCLAS 
will  reduce  the  validity  of  the  study. 

• Efficiency  of  operation  varies  with  workload  (busy 
or  slow  month) . During  summer  months,  laboratory 
workload  is  traditionally  low,  and  there  are  many 
military  transfers. 

• Short-term  perturbations  in  laboratory  operation 
(e.g.,  during  flu  season  or  over  Christmas  and 
New  Year  holidays)  will  influence  the  study. 

The  following  list  of  general  tasks  for  the  data  collectors 
provides  an  overview  of  the  data  collection  effort: 

• Gathering  workload  data  routinely  compiled  by 
laboratory  technicians 

• Counting  items  in  laboratory  files,  logs,  etc. 

• Performing  time  studies  of  laboratory  tasks 

• Observing  laboratory  activities  during  sampling 
periods 

• Sampling  records  and  reports 

• Administering  questionnaires 
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• Conducting  interviews 

• Gathering  cost  data  fron  the  finance  department. 

Generally,  specification  of  the  data  items  and  the 
collection  methods  (see  Appendix  B)  was  guided  by  the  hypo- 
theses and  the  associated  analyses.  These  are  discussed  in 
Sections  II  and  IV,  respectively.  In  addition,  information 
is  needed  to  assess  the  effects  of  intervening  factors,  and 
that  information  cannot  be  completely  specified  in  advance. 
Some  data  items  associated  with  the  hypotheses  will  be  useful 
in  studying  intervening  factors,  and  the  following  items  are 
included  specifically  for  that  purpose: 

• The  number  of  tests  per  day,  by  kind  of  test, 
received  from  any  facilities  other  than  out- 
patient clinics  and  dispensaries  that  routinely 
send  laboratory  specimens  to  the  clinical 
laboratory  for  analysis  in  Periods  X and  Y. 

• Significant  changes  in  the  laboratory  equipment 
configuration  (with  the  exception  of  AFCLAS 
hardware)  from  the  beginning  of  Period  X 
through  the  end  of  Period  Y. 

• Changes  in  administrative  policy  from  the 
beginning  of  Period  X through  the  end  of 
Period  Y,  regardless  of  source,  that  affect 
kinds  and  volume  of  tests  performed,  infor- 
mation handling  workload,  utilization  of 
expendables,  or  any  other  factors  being 
assessed  in  the  evaluation. 

In  response  to  a request  for  cooperation  from  the  Office 
of  the  Surgeon  General,  the  Directorate  of  Manpower  and 
Organization,  Deputy  Chief  of  Staff  Programs  and  Resources 
(AF/PRM)  assigned  members  of  the  Management  Engineering 
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Team  (MET)  to  collect  data  for  the  evaluation  of  AFCLAS  as 
described  in  Appendix  B The  MET  is  a group  of  officers  and 
noncommissioned  officers  (NCOs)  trained  in  industrial 
engineering  techniques.  Their  primary  task  is  to  support  base 
level  functional  managers  with  studies  of  manpower  related 
problems.  They  also  participate  in  establishing  and  main- 
taining manning  standards  for  diverse  work  centers  and 
functions.  The  base  level  MET  can  also  provide  management 
consultant  support  to  functional  managers.  In  this  capacity, 
they  exercise  a wide  range  of  investigative  and  analytical 
techniques  in  addressing  various  management  problems. 

The  data  collection  activities  have  been  designed  to 
interfere  minimally  with  personnel  at  the  medical  facility. 
However,  cooperation  and  some  participation  is  needed  in 
conjunction  with  the  tasks  of  the  MET  data  collectors. 

Appendix  B contains  lists  of  the  data  collection  activities 
that  require  assistance  from  laboratory  personnel  and  of  those 
that  require  the  assistance  of  other  facility  personnel. 
Assistance  of  laboratory  personnel  includes  distributing  forms 
to  a sample  of  patients,  completing  a questionnaire,  and  pro- 
viding daily  workload  records  to  the  data  collectors.  Tasks 
for  personnel  outside  the  laboratory  include  completing 
questionnaires  and  providing  certain  workload  information. 

One  task  requested  of  laboratory  officers  should  be 
mentioned  separately.  That  task  is  to  identify  changes, 
occurring  between  Periods  X and  Y,  that  are  not  specifically 
associated  with  AFCLAS,  and  any  other  intervening  factors  that 
could  potentially  affect  the  results  of  the  evaluation.  This 
is  particularly  important  since  the  data  collection  team  will 
not  be  onsite  during  the  interval  between  Periods  X and  Y. 

The  58  hypotheses  that  served  to  guide  the  study,  the 
methods  of  analyses,  and  the  methods  of  data  collection,  all 
imply  several  assumptions.  The  most  fundamental  assumptions 
are  as  follows: 
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• Changes  identified  by  the  hypotheses  are 
effects  of  implementing  AFCLAS. 

• The  hypothesized  changes  account  for  the  most 
significant  changes  due  to  AFCLAS. 

• The  case  mix  of  inpatients  seeking  help 
remains  constant. 

• The  proportion  of  each  beneficiary  class  for 
both  inpatients  and  outpatients  remains 
constant . 

it 

• The  values  of  several  variables  that  are 
affected  by  workload  vary  directly  with  the 
volume  of  tests  requested. 

• The  "Hawthorne  Effect"^  operates  during  both 
Period  X and  Period  Y,  and  the  net  impact  of 
the  Hawthorne  Effect  on  differences  between 
Period  X and  Period  Y is  zero. 

• During  Period  Y,  AFCLAS  operates  in  compliance 
with  the  RFP  specifications  and  terms  of  the 
contract . 

• The  unit  times  for  the  tasks  time  studied  remain 
constant  over  small  changes  in  workload. 

• The  unit  times  for  the  tasks  time  studied  remain 
constant  over  small  changes  in  staffing  levels. 


* This  assumption  was  made  when  adjusting  the  following 
variables  for  the  change  in  workload  between  Period  X 
and  Period  Y:  number  of  old  request  slips,  lines  on 

worksheets,  and  inquiry  phone  calls. 

t Working  extra  hard  because  of  the  feeling  of  participating 
in  something  new  and  special  has  come  to  be  known  as  the 
'Hawthorne  Effect'"  (Reference  1). 


21 


REFERENCES 


(1)  Edgar  H.  Schein.  Organizational  Psychology.  2nd  ed. 
Englewood  Cliffs,  New  Jersey:  Prentice-Hall , Inc., 

1970:  32. 


22 


IV.  ANALYSIS 


The  previous  sections  and  associated  appendices  list  58 
changes  as  hypothesized  effects  of  AFCLAS  to  be  evaluated, 
and  identify  the  data  to  be  collected  before  and  after  the 
system  is  installed.  This  section  describes  the  analyses  of 
that  data  that  will  be  performed  in  carrying  out  the  evaluation. 
Three  main  kinds  of  analyses  are  planned: 

• Statistical  studies  of  selected  hypothesized 
changes 

• Calculation  of  a net  cost  of  AFCLAS  using  system 
costs  and  the  impacts  that  can  be  quantified  in 
dollars 

• Evaluation  of  survey  questionnaire  responses. 

Separate  statistical  studies  will  be  made  of  anticipated 
changes  in: 

• Percentage  of  laboratory  reports  filed  in  the  out- 
patient medical  record 

• Percentage  of  laboratory  reports  filed  in  the 
inpatient  medical  record 

• Percentage  of  outpatient  request  slips  arriving 
at  the  reception  desk  with  one  or  more  serious 
errors 

• Duration  of  inquiry  phone  calls 

• Frequency  of  inquiry  phone  calls 

• Patient  time  in  clinical  laboratory  for  specimen 
collection 

• Patient  service  time  at  the  reception  desk 
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• Turnaround  time  for  routine  laboratory  reports 

• Turnaround  time  for  stat  laboratory  reports. 

All  changes  with  which  a dollar  benefit  or  cost  can  be 
associated  will  be  included  in  the  analysis  of  dollar  benefits 
and  costs  except  those  that  require  prohibitive  data  collection 
effort . 

Responses  to  survey  questionnaires  will  be  used  to  study 
the  acceptance  of  AFCLAS  by  the  several  categories  of  per- 
sonnel affected  by  it,  as  well  as  to  investigate  their  satis- 
faction with  the  system's  performance.  The  responses  will 
also  provide  information  for  evaluating  possible  effects  on 
patient  care  and  on  the  usefulness  and  timeliness  of  the 
laboratory  reports  that  are  provided. 

Some  effects  of  the  system  will  not  be  evaluated  in  these 
studies.  Some  of  them  have  been  specifically  excluded  as 
explained  in  Section  II.  The  proposed  studies  consider  a wide 
range  of  expected  impacts  and  should  also  reveal  any  signifi- 
cant ones  overlookedo 

The  proposed  analysis  is  described  in  the  following  sub- 
sections . 

A.  Accuracy  and  Completeness  of  Records,  Requests,  and  Reports 

1.  Completeness  of  the  Medical  Record 

A patient's  medical  record  is  not  complete  with  respect 
to  the  laboratory  work  that  has  been  done  for  him  unless  it 
contains  a report  of  all  tests  completed  for  him.  Medical 
record  completeness  is  therefore  defined  for  the  evaluation  as 
the  percentage  of  completed  laboratory  reports  that  are  filed 
in  the  medical  record  after  a specified  period  of  time.  A 
possible  benefit  of  AFCLAS  will  be  to  increase  medical  record 
completeness . 
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Inpatient  and  outpatient  records  will  be  studied  separately. 
In  each  case,  a sample  of  completed  reports  will  be  obtained 
from  the  laboratory  for  subs^'quent  followup  in  a medical  records 
search.  For  outpatients,  the  data  will  be  tested  for  changes 
between  Periods  X and  Y in:  a)  the  percentage  of  laboratory 

reports  found  in  the  medical  record  1 month  after  completion 
by  the  laboratory,  and  b)  the  time  between  completion  of  reports 
by  the  laboratory  and  their  arrival  in  the  Outpatient  Medical 
Records  Department.  For  inpatients,  a test  will  be  made  for  a 
change  in  the  percentage  of  reports  that  are  filed  in  inpatient 
records  on  the  wards  daily  for  5 days  after  they  are  completed 
in  the  laboratory  and  for  a change  in  the  percentage  that  are 
filed  in  inpatient  records  after  1 month. 

Since  laboratory  reports  and  laboratory  records  will  be 
different  before  and  after  the  automated  system  is  introduced, 
comparable  units  will  have  to  be  carefully  defined  for  sampling 
and  analysis.  As  an  illustration,  with  AFCLAS  a single  paper 
form  may  report  results  that  would  have  been  presented  on  three 
separate  forms  with  the  manual  system. 

2.  Errors  in  Arriving  Outpatient  Requests 

In  order  for  laboratory  reports  to  be  dis*-ributed  and 
filed,  the  laboratory  request  slip  must  contain  information 
identifying  the  patient  and  the  hospital  area  generating  the 
test  request.  Inpatient  identification  information  is  highly 
reliable  because  a plastic  or  metal  card  is  used  with  an 
addressograph  printer.  For  this  reason  errors  in  arriving 
inpatient  requests  are  not  studied  in  detail.  Outpatient 
identification  information  is  not  complete.  Therefore,  a 
statistical  test  for  a change  between  Period  X and  Period  Y 
will  be  conducted  with  respect  to  the  number  of  errors  in 
outpatient  request  slips  at  the  time  they  arrive  at  the 
reception  desk.  For  this  study,  such  errors  are  defined  as 
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missing  or  illegible  patient  name,  missing,  illegible  or 
incomplete  social  security  number,  and  illegible  or  missing 
clinic  name. 

3.  Accuracy  of  Test  Result  Reporting 

Improved  accuracy  of  test  result  reporting  is  expected 
after  AFCLAS  is  introduced  because  there  will  be  less  oppor- 
tunity for  errors  in  transcription  of  results  and  other  test 
information.  The  number  of  errors  avoided  will  be  roughly 
estimated  for  a sample  of  laboratory  sections  or  tests  by 
considering  the  numbers  of  transcriptions  required  before  and 
after  AFCLAS  is  introduced  and  using  standards  for  transcrip- 
tion error  rates. 

B.  Timeliness  of  Laboratory  Test  Results 

1 Laboratory  Request  Turnaround  Time 

An  important  objective  of  the  laboratory  is  to  provide 
results  of  laboratory  tests  to  physicians  as  quickly  as  possi- 
ble. Thus,  a reduction  in  request  turnaround  time  as  a result 
of  AFCLAS  would  be  a positive  contribution  to  the  operation 
of  the  laboratory.  Turnaround  time  begins  when  the  request 
is  received  by  the  laboratory  and  ends  when  the  report  of 
results  reaches  the  point  to  which  laboratory  personnel 
distribute  it.  (A  precise  definition  is  given  in  Appendix  A.) 
Turnaround  time  will  be  investigated  separately  for  stat  and 
routine  requests  and  by  laboratory  section  as  appropriate. 
Different  turnaround  times  are  characteristic  of  different 
sections  due  to  the  nature  of  the  task  performed.  The  data 
will  be  tested  for  a change  between  Periods  X and  Y in  the 
turnaround  time  for  these  types  of  requests. 

In  addition,  the  physician  questionnaire  will  provide 
information  about  the  physicians'  satisfaction  with  turnaround 
time  during  Periods  X and  Y and  whether  they  perceive  a change. 


26 


2. 


Telephone  Inquiries 


Another  potential  contribution  of  AFCLAS  to  timely 
availability  of  test  results  to  physicians  will  be  the 
increased  accessibility  of  the  information  to  laboratory 
personnel  when  answering  telephone  inquiries  and  accessi- 
bility of  results  in  clinics  and  wards  through  remote 
terminals  making  such  calls  unnecessary.  Thus,  data  on 
duration  and  frequency  of  such  calls  will  be  analyzed  for  a 
change  between  Periods  X and  Y. 

3.  Stat  Requests 

It  is  possible  that  the  percentage  of  tests  ordered 
stat  will  change  after  introduction  of  AFCLAS,.  In  the  event 
of  a decrease,  the  laboratory  will  experience  less  disrup- 
tion for  this  special  handling.  Part  of  the  additional 
technician  time  required  for  tests  ordered  stat  is  considered 
in  the  analysis  of  dollar  benefits  and  costs  that  are 
required  to  telephone  results  to  nursing  units  or  outpatient 
clinics — but  this  does  not  measure  the  total  impact  of  a 
change  in  the  proportion  of  stat  requests  so  the  percentage 
of  tests  ordered  stat  will  be  studied. 

4.  Remote  Terminals 

Timely  availability  of  test  results  will  be  enhanced 
where  remote  terminals  are  provided.  Through  the  use  of 
remote  terminals,  physicians  will  be  able  to  obtain  labora- 
tory results  before  they  receive  the  laboratory  report. 

C Dollar  Benefits  and  Costs 

The  objective  of  the  dollar  benefits  and  costs  analysis 
of  AFCLAS  is  to  estimate  the  net  change  in  laboratory  bene- 
fits and  costs  as  a result  of  introducing  AFCLAS;  that  is, 
to  estimate  the  cost  of  the  system  and  its  operation  decreased 
or  increased  by  the  cost  changes  associated  with  the  impacts 
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studied.  The  change  in  benefits  and  costs  to  be  derived  is 
the  difference  between  the  net  cost  of  operating  the 
laboratory  during  Period  Y with  AFCLAS  support  and  the  hypo- 
thetical benefits  and  costs  of  handling  the  Period  Y workload 
using  the  Period  X (manual)  methods.  The  cost  of  manual 
operation  will  be  estimated  using  the  workload  observed  during 
Period  Y and  the  data  on  manual  laboratory  operations  collected 
during  Period  X (e.g.,  what  tasks  are  performed,  and  with  what 
frequency  in  relation  to  test  workload) . If  this  net  cost 
is  negative  (implying  a dollar  savings) , the  nondollar  bene- 
fits of  the  system  are  "free."  If  the  net  cost  is  positive, 
it  must  then  be  weighed  against  the  other  benefits  not  quanti- 
fied in  dollars. 


Net  Change  in 
$ Costs  due 
to  AFCLAS 


Direct  $ Cost  ^ $ Change  in  Other  , . 

of  AFCLAS  Facility  Costs  ’ ^ 


In  addition  to  the  cost  of  the  AFCLAS  contract  for  pro- 
vision of  hardware,  software,  and  maintenance,  costs  of  the 
following  items  will  be  considered  in  calculating  the  direct 
dollar  cost  of  AFCLAS: 

• Site  preparation 

• Personnel  (assigned  exclusively  to  computer  operation) 

• Power  costs  (including  power  for  air  conditioning) 

• Maintenance  of  AFCLAS-related  facilities  and  equipment 
not  included  in  the  system  contract. 

The  greatest  change  in  facility  costs  will  most  likely 
be  a reduction  in  the  personnel  time  required  to  carry  out 
the  routine  clerical  tasks  of  operating  the  laboratory.  The 
change  in  personnel  cost  of  clerical  tasks  inside  the  clini- 
cal laboratory  is  designated  AC^.  Other  cost  changes  that 
will  be  included  in  the  analysis  are  changes  in  the  costs 


associated  with  certain  tasks  performed  outside  the  laboratory, 
designated  AC„ , and  changes  in  the  cost  of  paper  forms 

D 

(including  computer  forms)  designated  AC_. 

r 

Thus 


$ Change  in  Other 
Facility  Costs 


AC^  + ACg  + ACp  . 


The  methodology  is  the  same  for  calculating  both  the 
change  in  cost  AC^  of  personnel  time  inside  the  laboratory 
and  the  change  in  cost  of  personnel  time  outside  the  clinical 
laboratory  ACg.  The  methodology  for  calculating  AC^  is  to 
sum  for  all  tasks  that  were  time  studied,  the  difference 
between  the  cost  of  performing  the  task  in  Period  Y and  the 
cost  of  performing  the  task  in  Period  X.  For  any  task,  the 
difference  can  be  positive  or  negative.  A positive  value 
represents  an  increase  in  cost,  while  a negative  value  repre- 
sents a decrease  in  cost.  The  cost  for  a task  can  be  zero  in 
either  or  both  Periods  X and  Y.  The  costs  for  each  task  are 
subdivided  by  personnel  category.  Mathematically,  this  addi- 
tion of  cost  differences  is  expressed  by  equations  (ii)  and  (ii 

1.  Cost  Changes  for  Tasks  Inside  the  Laboratory 

In  this  case,  the  cost  change  to  be  estimated  (denoted  by 
AC^)  is  the  cost  of  laboratory  staff  time  saved  or  additional 
time  required  as  a result  of  task  changes  after  the  intro- 
duction of  AFCLAS  A "task  change"  for  the  laboratory  staff 
may  be  a change  in  the  number  of  times  the  task  must  be  done 
or  a change  in  the  procedure  for  performing  the  task.  The 
following  tasks,  for  which  changes  were  hypothesized,  will  be 
considered  in  estimating  AC^: 

• Producing  administrative  reports 

• Answering  telephone  inquiries 

• Filing  tasks  within  the  laboratory 
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• Preparing  laboratory  clinical  forms  (including  log 
books,  worksheet,  etc.) 

• Labeling  specimens 

• Preparing  specimen  collection  lists 

• Recording  and  transcribing  laboratory  test  results 

• Entering  test  requests,  patient  information,  and 
test  results  into  the  computer  system 

• Supervisors'  and  technicians'  reviews  and  certifi- 
cation of  results 

• Performing  statistical  analysis  for  quality  control 

• Performing  statistical  analysis  on  patient  results 
by  population 

• Calculations  and/or  conversions  from  raw  to  clinical 
values  for  test  reports 

• Operation  of  a computer  by  laboratory  staff  (does  not 
include  computer  operation  by  personnel  brought  into 
the  laboratory  to  perform  this  function) 

• Responding  to  inquiries,  complaints,  and  errors  by 
laboratory  supervisors 

• Reporting  results  of  8tat  tests  by  telephone 

• Reviewing  new  AFCLAS-generated  reports. 

The  cost  change  to  be  estimated  is  AC  , the  dollar  cost 
of  the  difference  between  the  laboratory  staff  time  required 
to  do  the  tasks  considered  during  Period  Y (with  AFCLAS)  and 
the  time  that  would  have  been  required  for  these  tasks  manually 
(without  AFCLAS) . The  cost  for  a task  can  be  zero  in  either 
or  both  Periods  X and  Y.  The  manual  time  requirements  are  to 
be  estimated  by  adjusting  Period  X time  requirements  to  the 
Period  Y workload.  AC^  is  the  sum  of  the  changes  observed 
for  the  individual  tasks. 
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(ii) 


n 

Z 

i=i 


where 


n 

AC. 


AC 


A. 

1 


the  number  of  tasks  included 

the  dollar  cost  change  in  manpower  for  task  i, 
as  a result  of  having  AFCLAS  during  Period  Y. 

= r / p<2'  y<2)  - p<l>  y.'l>  h<l>  1 = 

j=i  I 13  1 1 1 i : 


(iii) 


i=l,  2, 


n 


where 
k = 


c . = 
D 


number  of  laboratory  personnel  categories 
(laboratory  technician,  laboratory  officer,  etc.) 

cost  per  hour  of  personnel  category  j in  Period  Y 

number  of  units  of  task  i that  would  be  associated 
with  the  Period  Y workload  for  the  manual  system 
(t  = 1)  and  with  AFCLAS  (t  = 2) 


fraction  of  the  total  time  required  for  task  i that 
is  routinely  contributed  by  personnel  in  category 
j with  the  manual  system  (during  Period  X)  (t  = 1) 
and  after  introduction  of  AFCLAS  (during  Period  Y) 
(t  = 2)  . k ... 


number  of  hours  required  to  do  one  unit  of  task  i 
with  the  manual  system  (t  = 1)  and  with  AFCLAS  (t  = 2) • 


If  there  is  no  difference  between  the  number  of  task  i 

units  with  the  manual  system  and  with  AFCLAS,  then  y|^^  = y|^^ • 

Similarly,  if  the  time  required  per  unit  of  task  i is  the 

same  with  the  manual  system  and  with  AFCLAS,  then  hf^^=  hf^\ 

Values  of  the  variables  P — \ P — and  hf^^ 

■'1  •'1  1 1 

will  be  derived  from  data  collected  in  the  laboratory  before 
and  after  the  introduction  of  AFCLAS. 

AC^  should  be  adjusted  to  account  for  computer  downtime. 
Assuming  that  the  manual  system  is  used  during  computer  down- 
time, one  method  for  adjusting  the  value  for  AC^  would  be  to 


31 


reduce  the  absolute  value  of  each  sum  and  in  the  expressions 
for  AC^  by  the  actual  percentage  of  downtime.  This  adjusted 
value  i^  only  an  approximation  because  it  underestimates  the 
cost  for  downtime  in  the  prime  shift  by  ignoring  the  techni- 
cian time  involved  in  returning  the  laboratory  to  computer 
support  and  overestimates  the  cost  for  downtime  in  off  hours 
when  the  laboratory  is  running  with  a minimum  number  of 
technicians . 

The  value,  AC^,  as  it  has  been  described,  is  based  on  a 
comparison  of  laboratory  operations  during  two  periods  of 
data  collection.  The  effects  of  intervening  factors  such  as 
equipment  changes  (other  than  AFCLAS) , seasonal  variation  in 
the  laboratory  workload,  the  rate  of  change  of  the  laboratory 
workload,  and  the  rate  of  change  in  costs  will  be  considered 
to  generalize  and  project  the  results. 

2.  Cost  Changes  for  Tasks  Outside  the  Laboratory 

The  cost  changes  for  tasks  outside  the  laboratory  will 
be  estimated  in  the  same  manner  as  for  those  inside  the 
laboratory.  The  tasks  that  will  be  considered  are: 

• Placing  telephone  inquiries 

• Filing  laboratory  reports  in  outpatient  medical 
records  and  at  nursing  stations 

• Preparing  test  request  slips  outside  the  laboratory 

• Preparing  cumulative  laboratory  reports 

• Keyboard  entry  of  patient  information  in  the 
registrar's  office 

• Creating  for  laboratory  use,  a list  of  admissions 
and  interward  transfers. 

Equation  (iii)  will  be  used  to  calculate: 

ACg  = the  cost  change  in  laboratory-external  manpower 
dollars  of  having  AFCLAS  during  Period  Y. 
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3.  Change  in  the  Cost  of  Laboratory  Paper  Forms 

Some  change  in  the  cost  of  the  paper  forms  used  for 
laboratory  operations  may  result  from  the  introduction  of 
AFCLAS.  This  is  because  the  three-part  carbon  request 
slips  will  be  replaced  by  single-copy  mark-sense  cards  and 
because  many  laboratory  worksheets  and  log  books  will  be 
eliminated.  The  value  of  such  a cost  reduction  to  the 
laboratory  is: 

ACp  = the  change  in  paper  forms  cost  of  having  AFCLAS 
during  Period  Y (including  computer  forms) . 

ACp  is  the  difference  between  the  estimated  cost  of  using  the 
paper  forms  associated  with  the  manual  system  (those  in  use 
during  Period  X)  during  Period  Y,  and  the  actual  cost  of  paper 
forms  used  with  AFCLAS  during  Period  Y. 

4.  Dollar  Impact  of  Reports  Provided  by  AFCLAS 

AFCLAS  will  automatically  generate  some  of  the  reports 
currently  produced  by  manual  methods.  In  addition,  AFCLAS 
will  generate  new  reports  or  provide  additional  information 
on  old  ones.  The  benefit  expressed  as  cost  of  personnel  time 
of  replacing  manually  generated  reports  by  computer  generated 
reports  is  included  in  the  calculation  of  cost  changes  for 
tasks  inside  the  laboratory.  The  value  of  reports  not  avail- 
able prior  to  the  introduction  of  AFCLAS  is  difficult  to 
assess.  One  way  to  attach  a dollar  value  to  reports  generated 
by  AFCLAS  that  were  not  available  under  the  manual  system  is 
to  assign  them  a value  equal  to  the  cost  of  producing  them 
manually.  Such  a method  is  not  consistent  with  the  guidelines 
for  this  analysis.  While  it  is  reasonable  to  conclude  that 
the  reports  have  some  dollar  value  to  the  laboratory,  the  cost 
of  producing  them  manually  does  not  adequately  represent  that 
value.  The  implied  dollar  benefit  of  such  a report  probably 
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lies  somewhere  between  zero  and  the  cost  of  producing  it 
manually.  It  is  a function  of  how  frequently  the  report  is 
used  and  by  whom,  as  well  as  of  its  utility  to  the  users. 

An  estimate  of  the  dollar  benefit  would  be  the  amount  the 
laboratory  would  be  willing  to  spend  to  have  the  report 
without  AFCLAS.  However,  no  attempt  is  planned  in  this 
evaluation  to  obtain  these  hypothetical  amounts.  It  is  even 
more  difficult  to  assess  the  value  of  information  added  to 
reports  that  existed  previously.  The  usefulness  to  physicians 
of  new  reports  or  additional  information  on  old  reports  will 
be  studied  qualitatively  by  means  of  the  physician  schedule. 

In  addition,  reports  produced  with  the  manual  system  and 
those  produced  by  AFCLAS  will  be  listed  for  comparison. 

D.  Acceptance  and  Satisfaction 

Improved  satisfaction  with  the  clinical  laboratory  by 
persons  affected  is  a potential  benefit  of  AFCLAS.  This 
includes  physicians,  registered  nurses,  the  laboratory  staff, 
patients.  Outpatient  Medical  Records  staff,  and  Admissions 
and  Dispositions  Department  staff.  In  addition,  the  perception 
of  AFCLAS  by  these  people  will  be  a determinant  in  realizing 
other  aspects  of  the  system's  value.  Personnel  both  inside  and 
outside  the  laboratory  will  have  to  accept  new  and  changed 
procedures.  New  report  forms  must  be  acceptable  to  physicians 
as  well  as  to  laboratory  staff.  Satisfaction  by  physicians 
may  change  their  test  requesting  patterns  in  both  the  number 
and  kinds  of  procedures  requested. 

Scores  on  survey  questionnaires  administered  both  before 
(during  Period  X)  and  after  (during  Period  Y)  AFCLAS  is  intro- 
duced or  information  gained  during  interviews  will  be  used  as 
the  measure  of  acceptance  and  satisfaction  in  the  evaluation. 
The  following  groups  of  people  will  be  surveyed: 


34 


c 


• Physicians 

• Registered  nurses 

• Staff  of  the  laboratory  departments 

• Patients 

• Staff  of  the  Outpatient  Medical  Records  department 

• Staff  of  the  Admissions  and  Dispositions  department. 

The  same  method  will  be  used  for  scoring  both  the 
physicians'  and  registered  nurses'  questionnaires.  A method 
has  been  chosen  that  weights  each  question  differently,  since 
the  pretest  showed  that  the  respondents  attached  a wide  range 
of  importance  to  the  questions.  A standard  score  is  computed 
for  each  question  by  calculating  the  standard  deviation  of 
each  response  from  an  assumed  normal  distribution.  The 
laboratory  staff  questionnaire  is  made  up  of  two  parts.  The 
first  part  is  a conventional  job  satisfaction  questionnaire 
and  will  be  scored  using  a Likert  scale.  The  second  question- 
naire is  designed  solely  to  obtain  data  on  the  laboratory 
operation  so  it  will  not  be  scored.  The  responses  will  be 
tallied  and  summarized  in  a table.  The  laboratory  patient 
questionnaire  is  designed  to  measure  patients'  perceptions 
of  the  organizational  climate  of  the  clinical  laboratory  and 
will  be  scored  using  a conventional  Likert  scale. 

The  survey  questionnaires  and  further  description  of  the 
analyses  techniques  are  included  in  Appendix  C. 

E.  Benefits  to  Patients 

1.  Quality  of  Patient  Care 

The  laboratory  exists  to  support  patient  care,  but  the 
impact  of  the  laboratory  on  the  patient  is  not  direct  since 
it  occurs  through  its  usefulness  to  physicians.  Thus,  the 
impact  is  not  likely  to  be  perceived  by  the  patient.  Therefore, 
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the  physicians'  questionnaire  will  measure  a physician's 
perception  of  the  impact  of  the  laboratory  on  patient  care. 

2.  Patients'  Time  to  Obtain  Laboratory  Services 

The  time  it  takes  for  a patient  to  obtain  laboratory 
service  is  the  time  between  entering  the  laboratory  with  a 
request  slip  and  leaving  the  laboratory  after  all  necessary 
specimens  have  been  collected.  Data  on  the  time  patients 
spend  in  the  laboratory  will  be  tested  statistically  for  a 
change  between  Period  X and  Period  Y. 

F.  Demand  for  Laboratory  Services 

One  possible  effect  of  AFCLAS  could  be  a change  in  the 
demand  for  laboratory  services.  This  could  occur  in  at  least 
two  ways.  First,  as  previously  mentioned,  increased  satis- 
faction of  physicians  with  the  services  may  cause  them  to 
alter  their  patterns  of  test  requests.  Second,  increased 
patient  satisfaction  may  lead  to  greater  utilization  of  out- 
patient and  inpatient  services  and,  in  turn,  lead  to  increased 
demand  for  laboratory  services. 

Some  indication  of  a change  in  physician  utilization  of 
the  laboratory  may  be  obtained  from  an  examination  of  the 
ratios  of  outpatient  laboratory  requests  to  outpatient  visits 
and  inpatient  laboratory  requests  to  inpatient  bed-days, 
assuming  that  case  mix  does  not  change  and  that  the  distri- 
bution of  reasons  for  inpatient  and  outpatient  utilization 
does  not  change  significantly.  These  ratios  will  not  be 
reviewed  in  this  evaluation.  Historically,  the  workload  of 
clinical  laboratories  is  rising  for  a constant  patient  work- 
load. For  example,  when  the  physician  has  the  option  of 
ordering  a panel,  he  often  orders  a panel  rather  than  one  or 
two  tests.  Preliminary  analysis  of  historical  data  from  one 
test  site  indicated  an  increasing  workload  per  outpatient 
visit,  but  it  was  not  possible  to  estimate  accurately  the 
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actual  growth  rate  because  of  the  wide  fluctuation  in  the 
number  of  tests  requested.  Thus,  it  would  not  be  possible  to 
separate  a change  in  demand  for  laboratory  tests  due  to  AFCLAS 
from  a change  due  to  natural  fluctuations  and  an  overall  long- 
term increase  in  demand. 

Change  in  patient  use  of  outpatient  and  inpatient 
facilities  because  of  AFCLAS  will  not  be  investigated  because 
it  would  be  difficult  to  collect  valid  data  and  because  the 
expected  impact  of  AFCLAS  on  patient  demand  is  small. 

G.  Laboratory  Productivity  per  Technician 

One  way  that  the  changes  in  time  required  for  laboratory 
tasks  should  eventually  be  reflected  in  laboratory  operations 
is  in  a change  in  productivity  per  technician.  Such  a change 
will  be  determined  primarily  by  the  demand  for  laboratory 
tests  and  by  administrative  policies  and  decisions  concerning 
manpower  utilization.  The  effect  of  AFCLAS  on  the  laboratory's 
manpower  needs  would  probably  not  be  indicated  by  a measure 
of  the  laboratory's  productivity,  since  the  period  of  the 
evaluation  is  too  short.  Hence,  overall  laboratory  productivity 
will  not  be  studied. 

H.  Analysis  of  Intervening  Factors 

Changes  in  the  laboratory  that  are  not  related  to  the 
introduction  of  AFCLAS  will  probably  occur  in  the  interval 
between  Periods  X and  Y.  An  important  part  of  this  study  is 
to  identify  these  intervening  factors  and  to  investigate  their 
effects  on  the  hypothesized  changes.  Data  will  be  collected, 
and  laboratory  officers  will  be  asked  to  help  to  identify  inter- 
vening factors.  In  some  cases,  the  impact  of  intervening 
factors  on  hypothesized  changes  may  be  directly  measurable; 
in  some,  it  may  be  possible  to  estimate  the  magnitude  of  impact; 
in  others,  it  may  only  be  possible  to  note  that  an  external 
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change  took  place.  The  methods  of  analysis  to  study  interven- 
ing factors  will  be  dependent  on  the  changes  that  are  observed. 
Although  it  is  impossible  to  eliminate  the  effects  of  intervening 
factors,  careful  study  design  has  been  used  to  minimize  them 
wherever  possible. 
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V.  ESTIMATING  THE  EFFECTS 
OF  INTRODUCING  AFCLAS  AT  OTHER  FACILITIES 


One  of  the  goals  of  the  AFCLAS  evaluation  is  to  provide 
information  for  determining  whether  AFCLAS  should  be  prolif- 
erated. All  clinical  laboratories  using  a manual  information 
system  must  process  the  same  basic  information  in  about  the 
same  way  (i.e.,  test  requests  come  to  the  laboratory  on  request 
slips,  intermediate  and  final  answers  are  recorded  on  work- 
sheets, and  the  answers  are  returned  to  the  physician  on  the 
request  slip) . For  this  reason  it  is  probable  that  if  AFCLAS 
is  introduced  at  laboratories  of  approximately  the  same  size 
as  the  two  test  sites,  many  of  the  benefits  will  be  approxi- 
mately the  same  as  the  benefits  realized  at  the  two  test  sites. 
Such  an  extrapolation  of  benefits  is  less  justifiable  when 
estimating  the  expected  benefits  of  introducing  AFCLAS  into 
laboratories  significantly  smaller  or  significantly  larger 
than  the  two  test  sites.  Some  of  the  difficulties  include; 

• Laboratory  operation  is  more  likely  to  be  different. 

• The  numbers  and  kinds  of  automated  equipment  are 
more  likely  to  be  different. 

• The  numbers  and  kinds  of  automated  equipment  that 
will  be  online  with  AFCLAS  are  more  likely  to  be 
different. 

One  method  for  estimating  the  cost  benefits  of  introducing 
AFCLAS  into  another  Air  Force  clinical  laboratory  (call  it 
Laboratory  Z)  is  described  below. 

A large  portion  of  the  dollar  benefits  and  costs  of  intro- 
ducing AFCLAS  is  the  result  of  changes  inside  the  laboratory, 
such  as: 

• A reduction  in  the  number  of  laboratory  clinical  forms 
that  the  technicians  are  required  to  complete  (includes 
reduced  transcription) 
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• A reduction  in  filing  and  handling  of  laboratory 
forms 

• A reduction  in  the  amount  of  time  required  to 
complete  administrative  reports. 

With  a detailed  list  of  specifications  for  the  AFCLAS  system 
proposed  for  Laboratory  Z and  the  results  of  the  evaluation 
at  the  two  test  sites,  the  impact  of  AFCLAS  in  Laboratory  Z 
can  be  estimated.  A survey  of  the  operation  of  Laboratory  Z 
should  be  made  to  identify  the  following  prospective  effects 
of  AFCLAS: 

• All  laboratory  clinical  forms  that  will  be  eliminated 
by  AFCLAS 

• The  administrative  reports  that  will  be  compiled  by 
AFCLAS  rather  than  by  hand 

• Tasks  in  which  AFCLAS  will  reduce  the  filing  and 
handling  of  laboratory  forms. 

Once  these  changes  in  the  operation  of  Laboratory  Z are 
identified,  some  data  collection  will  be  necessary  to  estimate 
their  magnitude.  These  data  will  be  similar  to  the  data 
collected  at  each  test  site  during  Period  X. 

The  object  of  data  collection  at  Laboratory  Z is  to 
estimate  the  values  AC^  used  in  the  benefits  and  costs 
analysis.  Accurate  calculation  of  the  AC^  's  would  require 
data  that  can  only  be  collected  after  implementation  of  AFCLAS. 
However,  most  of  this  data  can  be  estimated  from  the  data 
collected  durinn  Periods  X and  Y at  the  two  test  sites.  The 
following  examples  are  given  for  clarity. 

• If  AFCLAS  will  eliminate  five  different  worksheets 
in  the  Chemistry  Department  of  Laboratory  Z,  then 

the  total  number  of  worksheets  that  will  be  eliminated 
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per  month  can  be  calculated.  Also,  the  average  time 
to  complete  one  worksheet  can  be  computed  using  data 
collected  during  Period  X.  Since  the  task  would  be 
completely  eliminated  by  AFCLAS,  all  necessary  data 
are  available  for  computing  AC^  for  the  task  of 
completing  chemistry  worksheets^' 

• The  time  required  to  retrieve  information  about  test 
status  for  the  physician  when  AFCLAS  is  implemented 
in  Laboratory  Z will  not  be  known  until  the  system 
is  actually  implemented.  However,  the  method  of 
retrieving  test  status  using  a CRT  terminal  will  be 
the  same  for  all  AFCLAS  systems.  Therefore,  the  time 
to  retrieve  information  about  test  status  can  be 
estimated  as  the  average  length  of  time  to  retrieve 
test  status  at  the  two  test  sites  in  Period  Y. 

The  new  tasks  created  by  introducing  AFCLAS  into  Laboratory 
Z will  be  essentially  the  same  as  those  identified  at  the  two 
test  sites.  The  time  involved  in  most  of  these  new  tasks  is 
directly  related  to  either  the  number  of  laboratory  tests 
processed  or  to  the  number  of  patients  having  laboratory 
tests.  Therefore,  the  increase  in  technician  time  spent  per- 
forming new  tasks  after  AFCLAS  is  installed  can  be  roughly 
estimated  from  data  collected  at  the  test  sites.  When  the 
dollar  cost  of  introducing  AFCLAS  into  Laboratory  Z is  known 
in  addition  to  the  previously  described  savings  and  costs,  an 
estimatea  net  cost  of  introducing  AFCLAS  can  be  calculated  as 
described  in  Section  IV-C. 

If  AFCLAS  is  accepted  and  satisfies  staff  members  at 
both  test  sites,  it  is  reasonable  to  assume  acceptance  and 
satisfaction  at  Laboratory  Z.  This  should  also  be  true  of  the 
other  nondollar  benefits.  If  the  evaluation  yields  different 
results  at  the  two  test  sites,  then  extrapolation  to  another 
clinical  laboratory  may  be  precluded. 


“.i 


Assuming  that  similar  results  are  measured  at  the  two 
test  sites,  as  briefly  outlined  above,  it  should  be  possible 
to  estimate  the  impact  of  introducing  AFCLAS  into  other  Air 
Force  laboratories.  At  the  present  time,  the  scope  of  this 
evaluation  does  not  include  estimating  the  impact  of  introduc- 
ing AFCLAS  into  any  of  the  candidate  Air  Force  laboratories; 
hence  a detailed  methodology  has  not  been  defined. 
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VI.  SUMMARY 


The  results  of  the  functional  evaluation  of  AFCLAS  are 
to  be  used  in  answering  three  questions: 

1.  What  is  the  value  of  AFCLAS  to  the  medical  facility 
studied? 

2.  What  would  be  the  value  of  AFCLAS  to  other  medical 
facilities? 

3.  What  changes  in  AFCLAS  could  be  made  to  enhance  its 
value  to  the  medical  facility  studied  or  to  others? 

The  evaluation  plan  primarily  addresses  the  first  question 
by  providing  for  calculation  of  the  net  dollar  cost  of  AFCLAS 
as  well  as  for  investigation  of  nondollar  benefits  of  the 
system.  The  plan  thus  recognizes  that  the  total  value  of 
AFCLAS  to  the  medical  facility  studied  cannot  be  expressed 
in  dollars. 

The  net  dollar  cost  of  AFCLAS  is  the  difference  between 
the  cost  of  processing  the  workload  of  the  clinical  laboratory 
with  AFCLAS  and  the  cost  of  processing  it  with  the  previously 
used  manual  system.  The  net  cost  may  be  negative,  indicating 
dollar  savings  after  introduction  of  AFCLAS.  The  following 
changes  are  hypothesized  and  are  considered  along  with  the 
direct  cost  of  AFCLAS  in  determining  the  net  cost:* 

• Changes  in  personnel  time  required  (both  inside  and 
outside  the  laboratory)  to  process  the  laboratory's 
workload  (this  includes  time  to  process  paperwork) ; 

• A change  in  the  kinds  and  numbers  of  paper  forms  used. 

*The  direct  dollar  cost  of  AFCLAS  includes  items  such  as 
equipment  cost,  installation  cost,  maintenance  cost,  computer 
personnel  salary  cost,  cost  of  electric  power  and  cost  of  air 
conditioning. 
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The  evaluation  of  nondollar  benefits  of  AFCLAS  includes 
surveys  or  interviews  to  study  the  reactions  of  persons  affected 
by  the  system  including  physicians,  patients,  the  laboratory 
staff,  and  staffs  of  nursing  units,  outpatient  clinics,  and  the 
Medical  Records  Department.  Also  to  be  investigated  is  the 
hypothesized  impact  by  AFCLAS  on: 

• The  proportion  of  tests  ordered  stat 

• The  amount  of  time  that  the  laboratory  staff 
must  work  in  addition  to  normal  duty  hours 

• The  completeness  of  the  medical  record 

• Timeliness  of  laboratory  results 

• New  reports  and  information  that  are  available 

• Patient  care. 

The  plan  was  formulated  to  include  as  many  of  the  potential 
effects  of  AFCLAS  as  possible  without  requiring  a prohibitive 
amount  of  data  collection.  Data  are  required  for  analysis  of 
hypothesized  changes  and  effects,  for  assignment  of  dollar 
costs,  and  for  investigation  of  factors  other  than  AFCLAS 
that  are  affecting  laboratory  operations.  These  data  include: 
numbers  and  distribution  by  kind  of  test  request,  time  require- 
ments of  various  tasks,  numbers  and  cost  of  personnel  by 
category,  costs  of  AFCLAS,  costs  of  paper  forms,  and  infor- 
mation about  equipment  changes  and  changes  in  administrative 
policies. 

The  answers  to  the  first  question  (What  is  the  value  of 
AFCLAS  to  the  medical  facility  studied?)  together  with  the 
data  on  current  operations  of  other  facilities  to  which  intro- 
duction of  AFCLAS  is  being  considered,  provide  a basis  for 
answering  the  second  question.  However,  using  the  data  to 
answer  the  second  question  is  not  planned  as  part  of  the 
evaluation. 


It  is  expected  that  the  data  collected  in  answering  the 
first  question  will  also  provide  a basis  for  identifying 
certain  changes  in  AFCLAS  that  might  increase  its  value. 
However,  much  of  the  data  needed  for  this  purpose  cannot  be 
anticipated  in  advance  of  its  installation.  More  specific 
evaluation  for  improvements  must  be  planned  when  those 
improvements  can  be  anticipated. 
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APPENDIX  A 
DEFINITIONS 


1.  Acceptance  - level  of  system  approval  as  measured  by 
scores  on  an  acceptability  instrument. 

2.  Administrative  reports  - reports  required  by  Headquarters 
USAF  or  subordinate  commands. 

3.  Cumulative  summaries  or  flow  sheets  - a summary  of  labora- 
tory test  results  displayed  so  that  it  is  easy  to  observe 
trends  over  time  in  the  results  of  any  specific  laboratory 
test , 


4.  Direct  dollar  cost  of  AFCLAS  - consists  of  operating  costs 
and  acquisition  costs.  The operating  costs  include  rental 
of  computer  hardware  and  auxiliary  equipment,  maintenance 
of  computer  and  auxiliary  equipment,  computer  supplies,  and 
electricity.  The  acquisition  costs  include  facility  modi- 
fication, acquisition  of  permanent  equipment,  shipping, 

and  installation  of  computer  hardware. 

5.  Hypothesis  - a tentative  assumption  about  a potential 
impact  of  introducing  AFCLAS.  Use  of  the  term  hypothesis 
does  not  imply  that  statistical  hypothesis  testing  will 
be  applied  in  each  case.  However,  where  appropriate, 
validity  of  the  hypothesis  will  be  tested  using  standard 
statistical  techniques.  The  hypotheses  are  intended  pri- 
marily to  serve  as  a guide  for  data  collection  and 
analysis . 


Intervening  factor  - a change  in  operation  or  policy  of 
either  the  laboratory  or  the  hospital  that  occurs  between 
the  beginning  of  Period  X and  the  end  of  Period  Y,  that 
was  not  associated  with  AFCLAS,  and  that  could  affect 
the  data  to  be  collected  for  evaluating  AFCLAS. 


7.  Laboratory  clinical  forms  - all  forms  (including  logbooks) 
used  in  the  laboratory  For  test  requests,  test  processing, 
and  test  reporting;  quality  control  forms  and  internal 
records  of  the  number  of  tests  performed. 

Laboratory  reports  - all  paper  forms  sent  to  physicians 
to  report  the  results  of  clinical  laboratory  tests. 

9.  Laboratory  request  - a paper  form  or  computer  card  sent 
to  the  clinical  laboratory  to  request  laboratory  tests. 

In  some  cases  a request  may  become  a laboratory  report 
when  results  are  written  on  it. 


10.  Laboratory  staff  - all  laboratory  personnel  excluding 
pathologists , laboratory  officers,  secretaries,  and 
volunteers . 


Prsc 
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11.  Quality  of  patient  care  - the  degree  to  which  health  care 
provided  by  a health  care  system  or  a specified  component 
of  that  system  meets  the  standards  or  norms  implicit  in 
the  system  or  explicitly  established  for  it. 

12.  Job  satisfaction  - level  of  overall  job  contentment  as 
measured  by  scores  on  a job  satisfaction  instrument. 

13.  Laboratory  technicians  - used  herein  to  include  both 
laboratory  technologists  and  laboratory  technicians. 

14.  Telephone  inquiry  - any  telephone  call  to  the  clinical 
laboratory  to  obtain  test  status  or  test  results. 

15.  Turnaround  time  - time  from  arrival  of  the  request  in 
the  laboratory  to  time  the  report  is  ready  for  distri- 
bution. The  specific  definition  of  turnaround  time 
varied  slightly  between  Period  X and  Period  Y and  between 
a routine  request  and  a stat  request  as  follows: 

Routine  reauest  Period  X - time  of  arrival  of 

, 

request  is  actual  time  of  arrival  for  all 
requests  that  arrive  between  0600  and  2000 
hours.  K constructive  time  of  0600  hours  is 
assigned  to  routine  requests  that  arrive 
between  2000  hours  and  0600  hours.  Completion 
time  occurs  when  reports  are  brought  to  the 
front  desk,  stamped  with  the  time,  and  placed 
in  a box  for  later  distribution. 

Routine  request  Period  Y - time  of  arrival  is 
actual  time  the  test  request  card  is  read  into 
the  AFCLAS  system  in  the  reception  area  for 
all  request  cards  that  arrive  between  0600  and 
2000  hours.  A constructive  time  of  0600  hours 
is  assigned  to  routine  requests  that  are  read 
into  the  system  between  2000  hours  and  0600 
hours.  Completion  time  is  the  time  daily 
cumulative  summaries  are  printed. 

'’-■at  reciuest  Period  X - time  of  arrival  of  request 
IS  actual  time  of  arrival  for  all  stat  requests. 
Completion  time  occurs  when  reports  are  brought 
to  the  front  desk,  stamped  with  the  time,  and 
placed  in  a box  for  later  distribution.  Note 
that  the  turnaround  time  defined  herein  is  turn- 
around time  for  paperwork.  In  most  cases  the 
results  of  stat  tests  are  telephoned  to  the 
physician  prior  to  bringing  the  report  to  the 
front  desk. 


50 


5*7*  request  Period  Y - time  of  arrival  is 
actual  time  the  test  request  card  is  read 
into  the  AFCLAS  system  in  the  reception  area. 
Completion  time  occurs  when  reports  are  printed 
on  the  stat  printer  immediately  following 
validation  of  the  results.  Note  that  the  turn- 
around time  defined  herein  is  turnaround  time 
for  paperwork.  In  most  cases  the  results  of 
stat  tests  are  telephoned  to  the  physician 
prior  to  validation  and  printing. 

Usability  of  laboratory  reports  - qualitative  assessment 
by  physicians  of  the  convenience  and  practicability  of 
reports . 
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APPENDIX  B 

DATA  COLLECTION  HANDBOOK 

Prepared  Prior  to  Period  X With  Minor 
Additions  Prior  to  Period  Y) 
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NOTE  ON  APPENDIX  B 


This  appendix  discusses  the  special  studies  involved  in 
the  evaluation  of  AFCLAS  and  describes  in  detail  the  data  to 
be  collected  as  well  as  the  methods  of  data  collection.  The 
data  collection  can  be  divided  into  the  following  categories 

• Measuring  or  estimating  the  time  required  to 
complete  specific  tasks  (work  measurement) 

• Counting  the  number  of  times  specified  items 
or  tasks  are  done 

• Identifying  the  personnel  involved  in  specified 
tasks  by  personnel  category 

• Obtaining  information  about  laboratory  operations 

• Obtaining  cost  information 

• Administering  survey  questionnaires. 

The  data  items  to  be  collected  and  the  methodology  for 
collecting  them  are  presented  in  this  appendix  in  the  form 
in  which  they  were  provided  to  Management  Engineering  Team 
(MET)  personnel  as  a data  collection  handbook. 

Discussion  of  the  questionnaires  is  deferred  until 
Appendix  C. 
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I. 


SUMMARY  OF  TIME  STUDIES  FOR  EVALUATION  OF  AFCLAS 


This  section  sununarizes  the  time  studies  for  the  evalu- 
ation of  AFCLAS,  along  with  their  associated  time  measurement, 
data  collection  item  to  be  counted,  and  hypotheses  (see 
Section  II  of  the  evaluation  plan)  . 

Associated 

Tas)c Time  Measurement  Items  Counted Hypotheses 


1.  Tasks  involved  in 
preparation  of 
administrative 
reports  and  College 
of  American  Pathol- 
ogists (CAP)  work- 
load reports 


2.  Filing  request 
slips  in  labora- 
tory files 

3.  Filing  test  work- 
sheets and  work- 
load log  sheets 
in  laboratory 
files  by  labora- 
tory section  and 
centrally 

4.  Filing  quality 
control  reports/ 
statistical 
summaries  in 
laboratory  files 

5.  Other  laboratory 
filing  tasks 

6.  Removing  out- 
dated items  from 
laboratory  files 
for  each  kind  of 
file 

7.  Filing  laboratory 
results  in  out- 
patient medical 
records 

8.  Filing  laboratory 
results  in  the 
medical  records 
of  inpatients 

(a)  on  wards 

(b)  in  medical 
record  room 


Time  per  task 
accompli  sliment 
for  each  task 


Time  per  page 


Time  per  report 
produced,  by 
laboratory 
section 


As  appropriate 


Time  per  task 
accomplishment 
by  kind  of  file 


Time  per  slip 


Number  of  times 
during  Period  X 
that  each  task  is 
done 


Test  worksheets 
and  workload  log 
sheets  by  labora- 
tory section 


Number  of  times 
reports  are 
produced  by 
laboratory 
section 

As  appropriate 


Number  of  times 
outdated  slips 
are  removed  from 
files,  by  kind 
of  file 

Request  slips  for 
outpatients 


Time  per  slip 

(a)  on  wards 

(b)  in  medical 
record  room 


Request  slips  ^or 
inpatients  and 

(a)  percent  filed 
on  wards 

(b)  percent  filed 
in  medical 
record  room 


Time  per  request  Reouest  slips 
slip 


1,  10 


6 


6 


6 


6 

6 


7 


8 
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Period  Y Period 


Task 


Time  Measurement  Items  Counted 


9,  Preparing  and 

filing  cumulative 
summaries  or 
flow  sheets  of 
patient  results 


Time  per  patient 


Percent 

inpatients  for 
whom  it  is  done 


10.  Filling  out  Time  per  slip  Request  slips 

request  slips 
at  clinics  and 
wards 


11.  Filling  in  Time  per  slip 

laboratory 
request  infor- 
mation on  slips 
by  laboratory 
personnel 


Number  of  slips 
and  percent 
filled  in  by 
laboratory 
personnel 


X 


12.  Entering  test 

results  on  slips 

(a)  printed 
automatical- 
ly 

(b)  transcribed 
from  work- 
sheets or 
log  books 

(c)  transcribed 
from  instru- 
ment display 

|(d)  transcribed 
from  print- 
out 


Time  per  test 
type  (a) 
type  (b) 
type  (c) 
type  (d) 


Number  tests  by 
kind 

(a)  kinds  type (a) 

(b)  kinds  type(b) 

(c)  kinds  type(c) 

(d)  kinds  type(d) 


(a)  online 

(b)  manually 
entered  into 
terminal  from 
worksheets 

(c)  mark-sense 
worksheets 

(d)  mark-sense 
computer  card 


13.  Entering  headings 
on  test  work- 
sheets, workload 
log  sheets,  log 
book  pages 


Time  per  sheet 
or  log  book 
page 


Test  worksheets, 
workload  log 
sheets,  and  log 
book  pages 


Associated 

Hypotheses 
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9,  51 


9 


9,  13,  14, 
18,  19,  29 


9,  29 
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14, 


Task 


Time  Measurement 


Items  Counted 


Associated 

Hypotheses 


Entering  patient 
identification 
and  tests  re- 
quested on  test 
worksheets,  work- 
load log  sheets 
and  in  log  books 
(except  Chemistry 
worksheets  at 
W-P) 

Time  per  encry  by 
type  of  sheet/log 
book 

Test  worksheet, 
workload,  log 
sheet,  and  log 
book  entries 

9,  29 

Entering  results 
on  test  work- 
sheets, workload 
log  sheets,  and 
log  books,  types 
(a)  , (b)  , (c)  , 

(d)  , in  12  above 

Time  per  test 
types  (a)  , (b)  , 
(c)  , (d)  , in  12 
above 

Worksheets,  work- 
load, log  sheets 
and  log  books 
types  (a) , (b) , 

(c) , (d) , in  12 

above 

9,  13,  14, 

Labeling 
specimen  tubes 

Time  per  tube 

Tubes  labeled 

11 

Preparing  "l:st" 
for  routine 
specimen  collec- 
tion trips 
(sorting  slips 
for  technicians 
who  collect  the 
spec imens ) 

Time  per  routine 
trip 

Number  of  routine 
trips  per  day  and 
number  of  tech- 
nicians making 
trip 

12 

Supervisor's  and 
technician ' s 
review  and 
certification 
of  results 

Time  per  test, 
request  slip, 
worksheet  or  other 
unit  certified 

Number  tests, 
request  slips, 
worksheets,  or 
other  units 
certified 

20,  21 

Performing 
statistical 
analysis  of 
quality  control 
sample  results 
and  recording 
results  on  report 

Time  per  task 
accomplishment  by 
laboratory  section 

Number  of  times 
statistics  are 
calculated  by 
each  laboratory 
section  and  by 
each  kind  of 
report 

22,  23,  24 

Task 


Time  Measurement 


I terns  Counted 


Associated 

Hypotheses 


Performing 
statistical 
analysis  of 
patient  results 
by  population 

Time  per  analysis 

Number  of 

analyses 

performed 

25,  26 

Performing  test 
result  calcula- 
tions or 
conversions 

Time  per  test  by 
kind 

Tests  by  kind 
for  which  calcu- 
lations are  done 

27 

Performing 
computer  oper- 
ation tasks 
(W-P) 

Time  per  task 
accomplishment 
for  each  task 

Number  of  times 
each  task  is 
done  (per  day, 
week,  or  month) 

28 

Calling  a ward 
or  clinic  to 
report  the 
results  of  a 
stat  test  (Con- 
sider calls 
originating  in 
stat  laboratory, 
hematology,  and 
urinalysis . ) 

Time  per  telephone 
call 

Number  of  tele- 
phone calls  to 
report  stat 
results 

56 

Supervisor ' s 
(laboratory 
officers) 
responding  to 
inquiries,  com- 
plaints, and 
errors 

Time  per 
supervisor 

As  appropriate 

53 

Processing 
inquiry  phone 
calls  to  the 
laboratory 

Time  per 

telephone 

call 

Number  of 
inquiry  calls 
to  the 
laboratory 

2,  3,  4 

Retrieval  of 
data  to  answer 
inquiries  for 
call  backs  and 
completing  the 
return  calls 

Time  per 

telephone 

inquiry 

Number  of 
inquiries 
requiring  a 
return  call 

3 
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Task 


Time  Measurement 


Associated 
Items  Counted Hypotheses 


27.  Reviewing  or 
processing  new 
AFCLAS-gene- 
rated  reports 

Time  per 
report 

Frequency  of 
each  report 

58 

28.  Addition  of 
Cytology  to 
AFCLAS  report- 
ing system 

Time  per 
cytology 
report 

Number  of 
reports 

11 

29.  Preparing  a 
list  of 

Time  per 
transaction 

Number  of 
admissions , 

57 

admissions  and  dispositions, 

interward  trans-  and  interward 

fers  for  labora-  transfers 

tory  use  with 
AFCLAS 


63 


II.  SPECIAL  STUDIES  IN  THE  EVALUATION  OF  AFCLAS 

This  section  details  the  procedures  to  be  used  in  data 
collection  for  a number  of  the  special  studies  in  the  evalu- 
ation of  AFCLAS. 

A.  Study  of  the  Completeness  of  the  Inpatient  Medical  Record 
and  the  Outpatient  Medical  Record  (Period  X) . 

This  study  is  made  by  observing  and  sampling  activities 
and  records  in  a specified  manner.  The  activities  observed, 
the  information  extracted  from  records,  and  the  procedures 
used  are  then  documented. 

Inpat, ent  and  outpatient  reports  will  be  treated 
separately  following  these  general  procedures.  Each  normal 
workday,  starting  on  the  first  day  of  data  collection  and 
continuing  for  16  days: 

1.  Xerox  all  the  completed  laboratory  reports  when 
they  are  ready  for  distribution.  (A  portable  Xerox 
3100  will  be  provided  in  the  laboratory  reception 
area . ) 

2.  Separate  the  Xerox  copies  of  the  completed  laboratory 
reports  into  the  categories  of  (1)  inpatient,  (2) 
outpatient,  (3)  outpatient  preoperative  (preop), 

and  (4)  tests  requested  from  remote  facilities. 

3.  At  the  end  of  the  day,  number:  (1)  the  Xerox  copies 

of  inpatient  laboratory  reports  sequentially  starting 
with  number  1,  (2)  the  Xerox  copies  of  outpatient 

laboratory  reports  sequentially  starting  with  number  1 
and  (3)  the  Xerox  copies  of  the  outpatient  preop 
reports  starting  with  number  1,  (4)  save  the  Xerox 

copies  of  reports  for  tests  requested  from  remote 
facilities,  but  do  not  consider  them  further  in  the 
study  of  completeness  of  the  medical  record. 
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4. 


After  the  reports  have  been  sequentially  numbered, 
draw  a random  sample  of  25  inpatient  laboratory 
reports,  25  outpatient  laboratory  reports,  and  10 
outpatient  preop  laboratory  reports.  (The  random 
sample  of  reports  is  drawn  for  16  days,  giving  a 
total  sample  size  of  400  inpatient  reports,  400 
outpatient  reports  and  160  outpatient  preop  reports.) 

5.  At  the  completion  of  the  study,  organize  the  Xerox 
copies  of  the  laboratory  reports  as  follows: 

• File  the  sample  of  400  inpatient  reports  together. 

• File  the  sample  of  400  outpatient  reports  together. 

• File  the  sample  of  160  outpatient  preop  laboratory 

reports  together. 

• File  the  copies  of  all  other  laboratory  reports 
together  by  date  Xeroxed. 

Inpatient  Medical  Records 

Each  normal  workday,  starting  on  the  first  day  of  data 
collection  and  continuing  for  16  days,  do  the  following: 

1.  For  each  inpatient  report  in  the  daily  random  sample 
of  25,  go  to  the  ward  where  the  inpatient  is  located, 
starting  about  1900  hours.  Look  for  the  laboratory 
report  in  the  following  places: 

• Physician's  in-basket  waiting  his  signature 
(applicable  to  Wr ight-Patter son  only) 

• The  area  where  reports  are  placed  until  they 
are  posted  in  the  patient's  medical  record 

• Patient's  medical  record. 


66 


2. 


For  each  report  found  in  step  1 above,  note  on  the 
copy  of  the  laboratory  report: 

• Where  the  report  was  found 

• Date 

• Approximate  time. 

3.  If  a report  was  not  found  in  step  1,  note  that  fact. 

4.  The  record  of  all  laboratory  reports  not  found  in 
the  medical  record  is  to  be  saved.  Look  for  each 
report  again  the  evening  of  the  next  normal  workday 
as  described  in  step  1 above.  Continue  the  search 
for  a given  report  for  5 normal  workdays  (7  calendar 
days)  or  until  it  is  located  in  the  patient's 
medical  record.  If  the  report  is  not  located  in 
the  patient's  medical  record  in  5 normal  workdays, 
discontinue  search  for  the  report,  but  retain  the 
Xerox  copy  for  the  month-end  check.  (This  study 
will  be  conducted  only  on  the  evenings  of  normal 
workdays.  Weekends  and  Federal  holidays  will  be 
ignored . ) 

5.  If  an  inpatient  is  discharged  before  a laboratory 
report  is  located  in  his  medical  record,  look  for 
the  report  on  the  ward  in  the  area  where  reports 

to  be  sent  to  Inpatient  Medical  Records  are  located. 
If  the  report  is  not  found  after  a total  of  5 normal 
workdays,  discontinue  search  for  the  report.  Retain 
the  Xerox  copy  for  the  month-end  check. 

6.  When  an  inpatient  is  transferred,  look  for  his 
laboratory  reports  both  on  the  patient's  current 
ward  and  on  his  previous  ward. 


s, 
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7.  Repeat  the  above  process  in  steps  1 through  6 for 
each  of  the  16  random  samples  of  25  reports  drawn 
on  the  first  16  normal  workdays  of  Period  X. 

Four  weeks  after  the  last  day  a sample  was  drawn,  take 
the  Xerox  copy  of  all  inpatient  reports  not  found  in  the 
patient's  medical  record  to  Inpatient  Medical  Records  or  to 
the  ward  if  the  patient  has  not  been  discharged.  Find  all 
medical  records  in  which  the  laboratory  reports  should  be 
filed.  Record  the  date  and  whether  the  laboratory  report  is 
in  the  patient's  medical  record  or  not  in  the  record. 

For  all  the  laboratory  reports  not  found  in  the  medical 
record,  determine  if  any  of  the  patient  identification  infor- 
mation on  the  Xerox  copy  is  missing,  illegible,  or  incomplete. 
Document  any  missing,  illegible,  or  incomplete  information. 

Outpatient  Medical  Records 

Collect  the  random  sample  of  25  laboratory  reports 
daily  for  16  days  (total  of  400  reports).  Four  weeks  after 
the  last  day  a sample  was  drawn,  search  the  appropriate  out- 
patient's medical  record  for  the  laboratory  reports  and  record 
the  date  and  whether  the  laboratory  report  is  in  the  patient's 
medical  record  or  not  in  the  record. 

When  looking  for  a specific  laboratory  report,  look  for 
the  patient's  medical  record  first  in  Outpatient  Medical 
Records.  If  a patient's  medical  record  is  checked  out,  note 
where  the  record  has  been  sent  and  also  determine  if  the 
laboratory  report  is  attached  to  the  medical  record  signout 
sheet.  If  the  laboratory  report  is  attached  to  the  signout 
sheet,  record  the  date  and  note  on  the  laboratory  report  that 
it  was  attached  to  the  signout  sheet.  If  the  medical  record 
is  checked  out  and  the  laboratory  report  is  not  attached  to 
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the  signout  sheet,  trace  the  patient's  medical  record  until 
it  is  found  and  record  the  date  and  whether  the  laboratory 
report  is  in  the  patient's  medical  record. 

Look  for  all  laboratory  reports  not  found  in  the 
patient's  medical  record  or  attached  to  the  medical  record 
signout  sheet  in  the  following  places  and  make  the  appropriate 
notation  when  a laboratory  report  is  located: 

• Clinics  that  keep  separate  medical  records  for  their 
patients  (e.g.,  at  Wright-Patter son  the  OB/GYN  clinic 
maintains  separate  records  for  women  having  prenatal 
care) . 

• The  area  where  Outpatient  Medical  Records  stores  non- 
filable  laboratory  reports.  For  all  nonfilable 
laboratory  reports,  determine  why  the  report  was 
nonfilable . 

• Any  other  appropriate  place  the  laboratory 
reports  might  be  stored  (e.g.,  physician's  incoming 
and  outgoing  baskets)  . 

For  10  normal  workdays  starting  the  first  day  of  Period  X, 
Xerox  all  laboratory  reports  arriving  at  Outpatient  Medical 
Records.  Record  the  date  the  report  was  Xeroxed.  This  is  for 
determining  the  interval  between  the  time  outpatient  reports 
are  ready  for  distribution  to  the  clinics  and  the  time  they 
arrive  at  Outpatient  Medical  Records. 

Outpatient  Preop  Laboratory  Reports  (Inpatient  Medical  Records) 

Outpatient  preop  laboratory  reports  (OPPOLRs)  are 
considered  separately  as  described  below.  An  outpatient  preop 
laboratory  request  is  a request  for  laboratory  work  for  an 
outpatient  who  will  be  admitted  to  the  hospital,  usually  later 
the  same  day,  for  surgery  the  next  day.  These  laboratory 
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reports  receive  special  processing  and  become  part  of  the 
patient's  inpatient  medical  record.  Completed  OPPOLRs  are 
usually  sent  to  the  Admissions  and  Dispositions  (A&D)  office 
in  the  afternoon.  When  the  patient  is  admitted  in  the  early 
evening,  his  laboratory  reports  should  be  in  the  A&D  office. 
Hence,  the  OPPOLR  becomes  part  of  the  patient's  medical 
record  at  the  time  he  is  admitted. 

Each  normal  workday  starting  on  the  first  day  of  data 
collection  and  continuing  for  16  days,  do  the  following  at 
the  same  time  the  reports  for  inpatients  are  processed: 

1.  Check  the  distribution  point  in  the  laboratory  at 
about  1900  to  determine  if  OPPOLRs  were  delivered 
in  the  afternoon.  If  reports  were  not  delivered, 
note  this  on  the  random  sample  of  10  reports. 
Determine  and  document  at  the  A&D  office  how  many 
of  the  patients  were  admitted. 

2.  For  each  OPPOLR  in  the  daily  random  sample  of  10, 
go  to  the  A&D  office  starting  about  1900  hours. 
Determine  the  following: 

• Whether  the  OPPOLR  is  in  the  A&D  office. 

Document  whether  the  OPPOLR  is  found  or  not  found. 

• Whether  the  patient  has  been  admitted.  If  the 
patient  has  been  admitted,  document  approximate 
time  of  admission  and  ward  location. 

3.  For  those  patients  who  have  been  admitted  and  whose 
OPPOLRs  were  not  in  the  A&D  office,  go  to  the  appro- 
priate ward  and  determine  and  document  the  following: 

• Whether  the  OPPOLR  is  posted  to  the  patient's 
medical  record. 
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• For  patients  whose  OPPOLR  is  posted  to  the 

medical  record,  whether  the  OPPOLR  was  posted  to 
the  medical  record  when  the  patient  arrived  on 
the  ward  or  whether  ward  personnel  had  to  look  for 
the  OPPOLR. 

4.  Repeat  the  above  process  in  steps  1 through  3 for 

each  of  the  16  random  samples  of  10  reports  drawn  on 
the  first  16  normal  workdays  of  Period  X 

B.  Study  of  Laboratory  Request  Turnaround  Time  Inside  the 
Clinical  Laboratory 

Laboratory  request  turnaround  time  inside  the  laboratory 
will  be  investigated  by  studying  the  time  between  arrival  of 
the  specimen  at  the  laboratory  and  the  time  the  laboratory 
report  is  ready  for  distribution.  This  will  be  accomplished 
by  stamping  requests  with  time  and  date  as  they  come  into  the 
laboratory  and  stamping  reports  with  time  and  date  when  they 
are  ready  for  distribution.  The  precise  definitions  for 
start  and  end  of  turnaround  time  will  be  different  for  each 
test  site  to  conform  with  the  procedures  that  exist  at  the 
site  for  time  and  date  stamping  of  laboratory  requests  and 
reports^  Also,  there  will  be  some  variation  between  Period  X 
and  Period  Y due  to  changes  in  certifying  reports  for  distri- 
bution and  availability  of  time  data. 

It  will  probably  be  necessary  to  determine  turnaround 
time  by  laboratory  section.  A sufficient  sample  size  will  be 
available  since  a copy  will  be  obtained  for  all  laboratory 
requests  and  reports. 

C.  Study  of  Telephone  Calls  to  the  Clinical  Laboratory 

Data  will  he  Collected  to  study  the  frequency  and  duration 
of  phone  calls  to  the  laboratory,  and  type  of  call  (inquiry  or 
non-inquiry) . In  addition,  the  frequency  of  other  calls  into 
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li. 


and  out  of  the  laboratory  on  the  phone  lines  used  by  the 
receptionist  will  be  monitored.  Wr iqht-Patter son  has  a 
decentralized  system  for  respondinq  to  inquiries  and  Andrews 
has  a centralized  system.  Therefore,  the  study  of  telephone 
calls  will  be  tailored  to  each  site. 

Wright -Pat ter son 

At  Wr iqht-Patter son  the  receptionist  answers  telephone 
calls  coming  into  the  laboratory  on  one  of  five  lines. 
Requests  for  test  results  or  test  status  are  transferred  to 
the  laboratory  section  that  performed  the  test.  Each  section 
maintains  records  of  the  tests  it  performed. 


Laboratory  reception  area  personnel  will  be  asked  to  keep 
a record  of  all  telephone  calls  received  during  the  second  and 
third  week  of  Period  X and  Period  Y,  Telephone  calls  in  each 
of  the  following  categories  are  to  be  counted: 

• Calls  completed  at  the  reception  desk  that  are 
inquiries  and  those  that  are  for  other  purposes. 

• Inquiries  and  other  calls  transferred  to  each 
laboratory  section  or  office  area. 

A form  will  be  provided  for  this  information. 


MET  personnel  will  collect  additional  data  on  telephone 
inquiries  during  eight  rantJomly  selected  15-minute  observation 
periods  per  day.  The  information  to  be  collected  includes: 


• Total  duration  of  inquiry  calls 
receptionist  time,  time  on  hold, 
technologist  - This  data  will  be 
the  light  of  each  telephone  line 
telephone . 


subdivided  into 
and  time  talking  with 
collected  by  watching 
on  the  receptionist's 


• Category  and 
collected  by 


location  of  caller  - This  data  will  be 
talking  with  reception  area  personnel. 
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• Disposition  of  calls  subdivided  into  calls  completed 
at  reception  area  and  those  transferred 

• Response  calls  - Inquiry  calls  transferred  to  laboratory 
sections  will  be  followed  up  after  the  observation 
period  to  determine  if  the  call  was  completed  or 
whether  a return  call  was  made  by  the  technologist. 

If  a return  call  was  made,  an  estimate  will  be  made 
of  the  time  to  retrieve  the  requested  information. 
Personnel  in  each  laboratory  section  will  be  asked  to 
record  all  response  calls  on  a form  provided  by  ANSER. 

• An  attempt  will  be  made  to  observe  the  duration  of 
calls  made  inside  the  laboratory  to  provide  a check  on 
estimated  duration  of  response  calls. 

A small  separate  study  will  be  conducted  by  MET  personnel 
to  determine  the  average  duration  of  a telephone  call  for  report- 
ing stzi  results.  Hematology,  3tat  Laboratory,  and  Urinalysis 
will  be  measured  separately  while  all  others  will  be  grouped 
together.  MET  personnel  will  also  conduct  a work  sampling 
study  to  determine  the  percentage  of  time  telephone  lines  are 
busy . 

Andrews 

Since  a central  file  of  all  laboratory  test  results  is 
maintained  in  the  reception  area  at  Andrews,  the  receptionist 
answers  most  inquiry  calls. 

Laboratory  reception  area  personnel  will  be  asked  to  keep 
a record  of  all  telephone  calls  received  during  the  second 
and  third  week  of  Period  X a, id  Period  Y.  Telephone  calls  in 
each  of  the  following  categories  are  to  bo  counted. 

• Calls  completed  at  the  reception  desk  that  are 
inquiries  and  those  that  are  for  other  purposes. 
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• Inquiry  and  other  calls  transferred  to  each  laboratory 
section  or  office  area, 

A form  will  be  provided  for  this  information. 

MET  personnel  will  collect  additional  data  on  telephone 
inquiries  during  four  randomly  selected  1/2-hour  observation 
periods  per  day.  The  information  to  be  collected  includes: 

• Total  duration  of  inquiry  calls  completed  at  the 
reception  desk 

• Response  calls  by  reception  area  personnel  - The 
duration  of  all  the  response  calls  initiated  by 
reception  area  personnel  during  the  observation 
period  will  be  measured,  and  an  estimate  will  be 
made  of  the  time  to  retrieve  the  requested  infor- 
mation. An  estimate  will  be  made  of  the  number 
and  duration  of  response  calls  initiated  in  the 
laboratory  sections. 

• Category  and  location  of  caller  - This  data  will 
be  collected  by  talking  with  reception  area 
personnel . 

• Disposition  of  calls  subdivided  into  calls  com- 
pleted at  the  reception  area,  calls  requiring 

a return  call,  and  calls  transferred. 

A small  separate  study  will  be  conducted  by  MET  personnel 
to  determine  the  average  duration  of  a telephone  call  for 
reporting  stat  results.  Hematology,  Stat  Laboratory,  and 
Urinalysis  will  be  measured  separately  while  all  others  will 
be  grouped  together.  MET  personnel  will  also  conduct  a work 
sampling  study  to  determine  the  percentage  of  time  telephone 
lines  are  busy. 
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D.  Study  of  Errors  in  Arriving  Outpatient  Laboratory  Request 
Slips 

Laboratory  request  slips  arriving  at  the  reception  desk 
are  to  be  screened  for  the  following  errors: 

• No  patient  name 

• Illegible  patient  name 

• No  social  security  number 

• Illegible  social  security  number 

• Incomplete  social  security  number 

• No  clinic  or  ward  name 

• Illegible  clinic  or  ward  name 

• No  date 

• Illegible  date 

• No  patient  telephone  number 

• Illegible  telephone  number 

• No  requesting  physician 

• Illegible  requesting  physician 

• No  rank  or  relation  code 

• Illegible  rank  or  relation  code 

• Other. 

For  analysis  and  the  determination  of  sample  size  a 
laboratory  request  slip  is  defined  to  be  in  error  if  one  or 
more  major  errors  exist  (the  first  seven  in  the  above  list) . 

A slip  is  defined  to  be  correct  if  all  requested  information 
is  on  the  slip  and  is  legible  or  if  it  contains  one  or  more 
minor  errors  'the  last  nine  in  the  above  list) . 


Two  1/2-hour  observation  periods  per  day  will  be  selected 
randomly  for  the  duration  of  Period  X and  Period  Y.  All  out- 
patient laboratory  request  slips  arriving  at  the  reception 
desk  during  the  observation  period  are  to  be  screened  for 
errors,  using  a data  collection  form  provided  by  ANSER.  Since 
the  possibility  exists  that  the  errors  on  two  or  more  request 
slips  brought  to  the  laboratory  by  one  person  at  the  same  time 
may  be  correlated,  the  analysis  will  probably  be  on  the  basis 
of  one  request  slip  per  patient.  For  this  reason  it  will  be 
necessary  to  record  the  patient's  initials  along  with  the  errors 
on  all  request  slips. 

If  feasible,  it  would  be  desirable  to  hand  out  the  patient 
questionnaire  at  the  same  time  laboratory  request  slips  are 
being  screened  for  errors.  A sample  size  of  200  patient 
questionnaires  will  be  required.  One  procedure  for  obtaining 
the  necessary  sample  size  would  be  to  hand  out  five  patient 
questionnaires  during  each  30-minute  observation  period,  then 
collect  all  patient  questionnaires  at  the  end  of  the  observa- 
tion period.  If  less  than  five  patients  come  in  during  an 
observation  period,  the  questionnaires  are  to  be  handed  out  at 
the  next  observation  period  in  addition  to  the  five  for  that 
period . 

E.  Service  Time  at  the  Laboratory  Reception  Desk  and  Total 
Patient  Time  at  ♦'he  Clinical  Laboratory 

These  two  studies  will  be  conducted  simultaneously  by 
observing  the  reception  area  during  three  randomly  selected 
1/2-hour  observation  periods  per  day  for  the  duration  of 
Period  X and  Period  Y. 

Service  time  at  the  laboratory  reception  desk  for  all 
patients  arriving  during  the  1/2-hour  observation  periods  is 
to  be  studied  by  collecting  data  on  time  of  arrival,  service 
at  the  reception  desk,  type  of  patient  (outpatient  or  inpatient), 
and  a note  if  the  patient  received  instructions  from  the 
receptionist. 


Total  patient  time  at  the  clinical  laboratory  for  all 
patients  arriving  during  the  1/2-hour  observation  periods 
is  to  be  studied  by  stamping  a data  collection  form  with  time 
and  date  when  the  patient  arrives  at  the  laboratory,  giving 
the  form  to  the  patient,  collecting  the  form  when  the  patient 
leaves  the  laboratory,  and  stamping  the  data  collection  form 
with  time  and  date  again  as  the  patient  leaves.  MET  personnel 
will  be  responsible  for  handing  out  the  data  collection  form 
and  insuring  that  the  receptionist  or  other  designated  person 
collects  the  form. 
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III. 


DATA  COLLECTION  FOR  EVALUATION  OF  AFCLAS 


This  section  details  the  data  that  will  be  collected  in 
the  evaluation  by  general  category  of  data. 

A.  Time  Measurement  and/or  Estimation 

This  subsection  specifies  the  data  to  be  collected  for 
determining  time  per  unit  that  is  spent  in  specified  tasks. 
The  data  are  to  be  collected  by  direct  observation  or  by 
interviews . 


1.  Time  per  task  accomplishment  for  each  task  involved 
in  producing  administrative  reports  and  College  of 
American  Pathologists  (CAP)  workload  reports 

Documentation — 

a.  A list  of  all  administrative  reports,  their 
frequency,  purpose,  and  who  requires  them 

b.  A list  and  description  of  all  tasks  involved 
in  preparation  of  administrative  and  CAP 
reports,  the  frequency  of  the  task,  and  copies 
of  the  forms  used 

c.  All  time  measurements  obtained,  a description 
of  the  measurement  procedures  used,  and  of 
any  calculations  done. 

2.  Time  per  request  slip  to  file  the  laboratory  copy  of 
the  slip  in  the  laboratory  files 

Documentation — 


a.  A description  of  the  filing  task  timed 

b.  A copy  of  each  kind  of  request  slip 

c.  All  time  measurements  obtained,  a description 
of  the  measurement  procedures  used,  and  of 
any  calculations  done. 
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PFECEDII'G  PA3£ 


BLANK-NOT  FIUt'.D 


1. 


Time  per  page  to  file  test  worksheets  and  workload 
log  sheets  by  each  laboratory  section  and  centrally. 
(Separate  determinations  may  be  needed  if  procedures 
vary  widely  among  locations.) 

Documentation — 

a.  A description  of  the  filing  procedures  timed, 
by  laboratory  section  and  centrally 

b.  A list  by  laboratory  section  of  all  kinds  of 
items  filed  and  a copy  of  each 

c.  All  time  measurements  obtained,  a description 
of  the  measurement  procedures  used,  and  of 
any  calculations  done. 

4.  Time  per  report  produced  to  file  quality  control 
records  and  statistical  summaries  in  the  laboratory 
sections  and  centrally 

Documentation — 

a.  A description  of  the  filing  procedures  timed 

b.  A list  of  the  kinds  of  reports  and/or 
summaries  filed  and  a copy  of  each  kind 

c.  All  time  measurements  obtained,  descriptions 
of  procedures  used,  and  of  any  calculations 
done . 

5.  Time  to  perform  other  laboratory  filing  tasks,  unit 
time  and  documentation  as  appropriate 

6.  Time  per  task  accomplishment  to  remove  outdated  items 
from  laboratory  files  for  each  kind  of  file  (in 
preceding  time  measurement  items  2,  3,  4,  and  5)  for 
which  task  is  done 
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Documentation — 


a.  List  of  all  kinds  of  files  included  in  the 
preceding  items  2,  3,  4,  and  5 for  which 
this  task  is  done 

b.  Description  of  each  procedure  timed 

c.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done 

7.  Time  per  request  slip  to  file  laboratory  results  in 
outpatient  medical  records 

Documentation — 

a.  Description  of  the  filing  procedure  timed 

b.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 

8.  Time  per  request  slip  to  file  laboratory  results  in 
the  medical  records  of  inpatients  on  the  wards  and 
time  per  request  slip  to  file  laboratory  results  in 
the  medical  records  of  inpatients  in  the  Medical 
Record  Room 

Documentation — 

a.  Description  of  the  filing  procedures  timed 

b.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 

9.  Time  per  patient  for  preparation  and  filing  in  the 
patient's  medical  record  of  cumulative  summaries  or 
flow  sheets  of  laboratory  results 
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Documentation — 


a.  Description  of  the  procedures  timed 

b.  All  time  measurements  obtained,  description 

of  the  procedures  used,  and  of  any  calculations 
done , 

10.  Time  per  request  slip  to  fill  out  requests  for 
laboratory  tests  at  the  clinics  and  wards 

Documentation — 

a.  Description  of  the  procedure  timed 

b.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 

11.  Time  per  request  slip  spent  by  laboratory  personnel 
in  filling  out  or  completing  request  information  on 
laboratory  slips 

Documentation — 

a.  Description  of  procedure  timed 

b.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 

12.  Time  per  test  to  enter  test  results  on  laboratory 
slips 

For  Period  X: 


• 

For 

test 

results 

printed  automatically 

• 

For 

test 

results 

transcribed 

from 

worksheets 

or 

log  books 

• 

For 

test 

results 

transcribed 

from 

an  instrument 

display 
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• For  test  results  transcribed  from  a machine 
printout . 

For  Period  Y; 

• Online 

• Manually  entered  into  terminal  from  worksheets 

• Mark-sense  worksheets 

• Mark-sense  computer  card. 

Documentation — 

a.  Description  of  the  procedures  timed  (See 
Section  B,  number  7,  and  Section  E,  number 
4 for  list  of  tests.) 

b.  All  time  measurements  obtained,  description 
of  procedures  used,  and  description  of  any 
calculations  done. 

13.  Time  per  page  to  enter  headings  on  each  kind  of  test 
worksheets,  workload  log  sheets  and  log  book  pages 
(except  those  that  are  computer-generated  pages, 
Wright-Patter son  only) 

Documentation — 

a.  Descriptions  of  procedures  timed 

b.  List  of  all  kinds  of  log  books,  test  worksheets, 
workload  log  sheets,  and  a copy  of  each  kind 

of  page  that  headings  are  entered  on 

c.  All  time  measurements  obtained,  description  of 
procedures  used,  and  of  any  calculations  done. 

14.  Time  per  entry  (line,  patient)  to  record  patient 
identification  and  test  requests  for  each  kind  of 
test  worksheet,  workload  log  sheet,  and  log  book 
(except  those  that  are  computer-generated,  Wright- 
Patterson  only) . 
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Documentation — 

a.  Descriptions  of  the  procedures  timed 

b.  List  of  all  kinds  of  tes^  woxksbvee^-s  > 

wgrJcLoad — iTjri'  sTieets  and  log  books,  and 

^ a copy  of  each  kind  of  page  that  patient 

identification  and  requests  are  entered 
on 

c.  All  time  measurements  obtained,  descrip- 
tion of  procedures  used,  and  of  any 
calculations  done. 

15.  Time  per  test  to  enter  test  results  on  test  worksheets, 

workload  log  sheets,  and  log  books 

• For  test  results  printed  automatically 

• For  test  results  transcribed  from  other 
worksheets  or  log  books 

• For  test  results  transcribed  from  an  instru- 
ment display 

• For  test  results  transcribed  from  a machine 
printout . 

Documentation — 

a.  Description  of  the  procedures  timed 

b.  List  of  all  kinds  of  test  worksheets,  work- 
load log  sheets  and  log  books,  and  a copy 
of  each  kind  of  page  that  test  results  are 
entered  on 

c.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 
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16.  Time  per  tube  to  label  specimens 
Documentation — 

a.  D-’scription  of  the  procedures  timed 

b.  All  time  measurements  obtained,  descriptions 
of  procedures  used,  and  of  any  calculations 
done . 

17.  Time  per  routine  (scheduled)  specimen  collection  trip 
to  sort  slips,  prepare  a list,  or  otherwise  specify 
the  patients  to  be  visited  by  each  technician 

Documentation — 

a.  Description  of  the  procedures  timed 

b.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 

18.  Time  per  test,  request  slip,  worksheet,  or  other  unit 
reviewed  and/or  certified  for  review  and  certification 
of  test  results  by  supervisors,  and  time  per  test, 
request  slip,  worksheet,  or  other  unit  reviewed  and/or 
certified  for  review  and  certification  of  test  results 
by  technicians. 

Documentation — 

a.  List  of  each  kind  of  unit  reviewed  and/or 
certified  by  supervisors,  each  kind  of  unit 
reviewed  and/or  certified  by  technicians 

b.  Description  of  the  procedures  timed 

c.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 
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19.  Time  per  analysis  to  perform  statistical  analyses  of 
quality  control  sample  results,  or  other  calculations 
for  quality  control  and  to  record  them  on  a report 

Documentation — 

a.  Description  of  procedures  timed 

b.  List  of  all  kinds  of  quality  control  analyses 
performed  and  a copy  of  a report  for  each  kind 
(See  Section  A,  number  4,  documentation  item  b.) 

c.  All  time  measurements  obtained,  description 

of  procedures  used,  and  of  any  calculations  done. 

20.  Time  per  analysis  to  perform  statistical  analyses  of 
patient  results  by  population 

Documentation — 

a.  Description  of  procedures  turned 

b.  List  of  all  kinds  of  analyses  performed  and 
a copy  of  a report  of  each  kind 

c.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 

21.  Time  per  test  for  test  result  calculations  and/or 
conversions 

Documentation — 

a.  Description  of  procedures  timed 

b.  List  of  all  tests  for  which  calculations  and/or 
conversions  are  done  and  description  of  the 
calculation  and/or  conversion  required 

c.  All  item  measurements  obtained,  description  of 
the  procedures  used,  and  of  any  calculations 
done. 


■T> 
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22. 


Time  per  task  accomplishment  for  computer  operation 
tasks  such  as  generating  test  worksheets  with  CREATE, 
a time-sharing,  computer  system  used  at  Wright- 
Patterson 

Documentation — 

a.  List  of  computer  operation  tasks 

Descr -i  pt  i nr.  of— procedures  timed 

c.  All  time  measurements  obtained,  description 
of  procedures  used,  and  of  any  calculations 
done . 

23.  Time  per  telephone  call  to  report  stat  results 
Documentation — 

a.  Description  of  procedure  timed 

b.  All  time  measurements  obtained,  descriptions 
of  procedures  used,  and  of  any  calculations 
done . 

24.  Time  per  supervisor  to  respond  to  inquiries,  complaints, 
and  errors 

Documentation — 

a.  Description  of  procedures  timed 

b.  All  time  measurements  obtained,  descriptions 
of  procedures  used,  and  of  any  calculations 
done . 

25.  Time  per  admission  or  interward  transfer  of  admissions 
and  interward  transfers 

Documentation — 

a.  Description  of  procedures  timed 

b.  All  time  measurements  obtained,  descriptions  of 
procedures  used,  and  of  any  calculations  done. 
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26.  Time  per  report  to  review  or  process  new  AFCLAS- 
generated  reports 

Documentation — 

a.  Description  of  procedures  timed  

b.  —A-ll  L-inTS"  measurements  obtained,  descriptions 

of  procedures  used,  and  of  any  calculations 
done . 

B.  Counting  and/or  Estimation 

The  measurements  are  made  by  counting  or  estimating  the 
number  of  specified  items  and  tasks  done  during  Period  X and 
Period  Y;  the  numbers  obtained  and  the  procedures  used  are 
then  documented. 

1.  The  number  of  times  during  Period  X and  Period  Y 
that  each  task  involved  in  producing  administrative 
reports  and  CAP  workload  reports  is  performed 

Documentation — 

a.  See  Section  A,  number  1,  documentation  items 
a and  b. 

b.  All  numbers  obtained,  description  of  pro- 
cedures used,  and  of  any  calculations  done. 

2.  The  number  of  request  slips  received  by  the  laboratory 
each  day,  by  laboratory  section  and  by  inpatient  and 
outpatient  categories,  further  subdivided  by  stat 

and  all  others  («ount  stat  slips  only  in  Stat 
Laboratory,  Urinalysis,  and  Hematology.) 

Documentation — 

a.  List  of  kinds  of  slips  received  by  each 
laboratory  section  and  a copy  of  each  one 
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b.  All  daily  slip  counting  forms  and  a description 
of  the  procedures  used  to  obtain  the  numbers 
recorded . 

3.  The  number  of  test  worksheets,  workload  log  sheets, 
and  log  book  pages  of  each  kind  filled  out  each  day 
by  each  laboratory  section 

Documentation — 


a.  List  of  each  kind  of  test  worksheet,  work- 
load log  sheet,  and  log  book  used  by  each 
laboratory  section  and  a copy  of  each  kind 
of  page  (includes  hand-prepared  pages  and 
a copy  for  each  kind  of  use  of  a standard 
form) 

b.  All  daily  forms  for  counting  the  pages  and 
description  of  procedures  used  to  obtain 
the  numbers  recorded. 

4.  The  number  of  test  worksheets,  workload  log  sheets 
and  log  book  entries  per  day  by  kind  of  sheet  or 
log  book 

Documentation — 


a.  See  Section  B,  number  3,  documentation  item  a, 

b.  All  numbers  obtained,  a description  of  the 
procedures  used,  and  of  any  calculations  done. 

5.  The  number  of  times  during  Period  X and  Period  Y that 
each  kind  of  statistical  analysis  for  quality  control 
is  performed  and  that  each  kind  of  quality  control 
report  or  statistical  summary  is  produced  by  each 
laboratory  section 

Documentation — 
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a.  See  Section  A,  number  4,  documentation  item  b, 
and  number  19,  documentation  item  b. 

b.  All  numbers  obtained,  description  of  pro- 
cedures used,  and  description  of  any  calcula- 
tions done. 

6.  The  number  of  times  outdated  items  are  removed  from 
each  kind  of  laboratory  file  (those  for  which  the 
time  to  do  this  task  is  measured.  Section  A,  number  6) 

Documentation — 


a.  See  Section  A,  number  6,  documentation  item  a. 

b.  All  numbers  obtained,  a description  of  the 
procedures  used,  and  of  any  calculations  done. 

7.  The  number  of  tests  done  each  day  by  kind  of  test, 

( further  subdivided  into  inpatient  and  outpatient  for 

all  tests  and  then  by  routine  and  stat  for  tests  per- 

formed by  the  Stat  Laboratory,  Hematology  and  Urinalysis 

Dc jumentation — 

a.  List  of  all  kinds  of  tests  done  by  each 
laboratory  section 

b.  All  daily  test  counting  forms  and  description 
of  procedures  used  to  obtain  the  numbers 
records . 

8.  The  number  of  specimen  tubes  labeled  per  day 

’ Documentation — 

a.  All  numbers  obtained,  a description  of  the 
procedures  used,  and  a description  of  any 
calculations  done. 
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9. 


The  number  of  routine  specimen  collection  trips 
per  day  and  the  number  of  technicians  involved 
in  each 


Documentation — 

a.  All  numbers  obtained,  a description  of  the 
procedures  used,  and  of  any  calculations 
done . 


10.  The  number  per  day  of  each  kind  of  item  reviewed 
and  certified  by  supervisors  and  by  technicians 
(probably  included  in  the  preceding  items  of 
Section  B) 

Documentation — 

a.  See  Section  I,  number  18,  documentation 
item  a. 

b.  All  numbers  obtained,  a description  of 
procedures  used,  and  a description  of  any 
calculations  done. 

11.  The  number  of  times  during  Period  X and  Period  Y 
that  statistical  analysis  of  patient  results  by 
population  are  done 

Documentation — 


a.  See  Section  I,  number  20,  documentation 
item  b. 

b.  All  numbers  obtained  and  a description  of  the 
procedures  used. 

12.  The  number  of  times  per  day  each  computer  operation 
task  is  carried  out  (Wr ight-Patterson  only) 

Documentation — 
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a.  See  Section  I,  number  22,  documentation 
item  a. 

b.  All  numbers  obtained  and  a description  of 
the  procedures  used  to  obtain  them. 

13.  The  number  of  tests  sent  to  other  facilities  for 
processing  during  Period  X and  Period  Y by  kind  of 
test  and  the  facility  it  is  sent  to 

Documentation — 

a.  A list  of  all  kinds  of  tests  that  are 
routinely  sent  to  other  facilities 

b.  A list  of  all  facilities  that  the  tests 
are  sent  to 

c.  All  completed  "tests  sent  out"  counting 
forms  and  a description  of  the  procedures 
used  to  obtain  the  information  recorded. 

14.  The  number  of  tests  received  from  other  facilities 
during  Period  X and  Period  Y by  kind  of  test  and 
sending  facility 

Documentation — 

a.  A list  of  all  facilities  that  send  in  tests 
for  processing 

b.  A list  of  all  tests  that  the  sending  facili- 
ties routinely  send 

c.  All  completed  "tests  sent  in"  counting  forms 
and  a description  of  the  procedures  used  to 
obtain  the  information  recorded. 

15.  The  daily  inpatient  census  by  ward 
Documentation — 
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a.  List  of  all  wards  and  the  type  of  patient 
on  each  (e.g.,  Orthopedic,  Obstetrics) 

b.  All  daily  census  data  collected  and  a 
description  of  the  procedures  used  to  obtain 
it. 

16.  The  number  of  outpatient  visits  per  day  by  clinic 
Documentation — 

a.  List  of  all  clinics  and  type  of  clinic 

b.  All  daily  patient  visit  data  collected 
and  a description  of  the  procedures  used 
to  obtain  it. 

17.  The  number  of  telephone  calls  per  day  to  report 
stat  results 

Documentation — 

a.  List  sections  placing  calls  to  report  stat 
results 

b.  All  numbers  obtained  and  a description  of 
the  procedures  used  to  obtain  them. 

18.  For  each  new  AFCLAS-generated  report,  the  number  of 
times  per  week  the  report  is  printed 

Documentation — 

a.  List  all  reports  and  a copy  of  each  kind  of 
report 

b.  All  numbers  obtained  and  a description  of  the 
procedures  used  to  obtain  them. 


93 


c 


Personnel 


It  is  necessary  to  identify  the  personnel  involved  in 
each  of  the  following  tasks  and  to  determine  the  fraction  of 
the  total  time  for  the  task  that  each  contributes.  Procedures 
and  findings  are  documented  for  each. 

1.  Each  task  involved  in  preparation  of  administrative 
reports  and  CAP  workload  reports 

2.  Filing  laboratory  copy  of  request  slips  in  laboratory 
files 

3.  Filing  test  worksheets  and  workload  log  sheets  in 
laboratory  files — in  laboratory  sections  and 
centrally 

4.  Filing  quality  control  reports  and  statistical 
summaries  in  laboratory  central  files  and  in  labora- 
tory section  files 

5.  Removing  outdated  items  from  files  for  each  kind 
of  file  in  preceding  items  2,  3,  and  4 

6.  Filing  laboratory  results  in  Outpatient  Medical 
Records 

7.  Filing  laboratory  results  in  medical  records  of 
inpatients 

• On  wards 

• In  the  Inpatient  Medical  Records  Room 

8.  Preparing  cumulative  summaries  and  flow  sheets  of 
patients'  laboratory  results 

9.  Filling  out  request  slips 

• On  wards 

• In  clinics 

• In  the  laboratory 
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10.  Entering  laboratory  test  results  on  request  slips 
for  each  laboratory  section 

11.  Entering  headings,  patient  identification  requests, 
and  results  on  each  kind  of  test  worksheet, 
workload  log  sheet,  and  log  book 

12.  Labeling  specimens  (or  specific  tasks  involved — 
writing  name,  batch  labeling,  etc.) 

13.  Preparing  list  or  other  identification  of  patients 
to  be  visited  by  each  technician  on  specimen 
collection  rounds 

14.  Supervisor's  review/certification  of  results 

15.  Technician's  review/certification  of  results 

16.  Statistical  analysis  of  quality  control  results 

17.  Statistical  analysis  of  patient  results  by  population 

18.  Test  result  calculation  and/or  conversion  by  kind 

19.  Each  computer  operation  task  (Wright-Patter son  only) 
Documentation — 

a.  A list  of  name  and  classification  of  all 
laboratory  personnel 

b.  A description  of  the  procedures  used  to 
obtain  the  information. 

20.  Calling  a ward  or  clinic  to  report  the  results  of  a 
stat  test 

21.  Supervisor  (laboratory  officer)  responding  to 
inquiries,  complaints,  and  errors 

22.  Preparing  a list  of  admissions  and  interward  trans- 
fers for  laboratory  use 

23.  Reviewing  or  processing  new  AFCLAS-generated  reports. 
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D.  Information  About  Laboratory  Operations 

Specified  information  and  documentation  about  laboratory 
operations  must  be  obtained  from  the  personnel  involved. 

1.  Identify  and  list  any  records  of  laboratory  results 
(other  than  request  slips,  test  worksheets,  workload 
log  sheets,  quality  control  reports  and  statistical 
summaries)  that  are  filed  in  laboratory  files. 

2.  Identify  and  list  locations  in  the  laboratory  where 
telephone  inquiries  are  routinely  referred. 

3.  Obtain  and  estimate  the  percent  of  inpatient  labora- 
tory slips  filed  that  are  filed  on  the  wards  and  the 
percent  filed  in  the  medical  record  room,  and 
describe  procedures  used. 

4.  List  all  laboratory  tests  in  the  following  way: 

• Tests  for  which  results  are  printed  on  request 
slips  automatically 

• Tests  for  which  results  are  transcribed  to  slips 
from  worksheets  or  log  books 

• Tests  for  which  results  are  transcribed  to  slips 
from  an  instrument  display 

• Tests  for  which  results  are  transcribed  to  slips 
from  a machine  printout. 

5.  List  tests  for  which  results  are  entered  on  test 
worksheets,  workload  log  sheets,  or  log  books  in  the 
following  way: 

• Those  printed  automatically  on  the  worksheet, 
workload  sheet,  or  log  book 

• Those  transcribed  from  other  worksheets,  workload 
sheets,  or  log  books 


• Those  transcribed  from  an  instrument  display 

• Those  transcribed  from  a machine  printout 

6.  Identify  and  list  any  laboratory  forms  used  in 
requesting  tests,  processing  tests,  reporting  test 
results,  and  keeping  quality  control  records  that 
have  not  been  included  in  the  preceding  lists. 

7.  Identify  and  list  all  automated  laboratory  equipment 
and  the  tests  done  by  each.  List  those  that  will  be 
online  with  AFCLAS  and  those  that  will  be  offline. 

8.  For  each  laboratory  section  list  the  kinds  of  quality 
control  statistics  that  are  calculated. 

9.  Identify  and  list  all  tests  for  which  technicians  do 
calculations  and/or  conversions  and  specify  the 
calculation  and/or  conversion  done. 

10.  Obtain  estimates  from  laboratory  supervisors  of  the 
time  they  spend  in  responding  to  inquiries,  complaints, 
errors  detected  before  results  leave  the  laboratory, 
and  errors  in  results  reported  to  physicians.  Describe 
the  procedures  used. 

11.  Obtain  an  estimate  of  the  percent  of  laboratory  request 
slips  for  which  request  information  is  filled  in  at 
the  laboratory.  Describe  the  procedures  used. 

12.  Obtain  an  estimate  of  the  percent  of  inpatients  for 
whom  cumulative  summaries  or  flow  sheets  of  laboratory 
results  are  prepared. 

f 

E.  Cost  Data 

f Cost  information  will  be  collected  on  the  items  listed  below: 

1.  Identify  total  cost  of  AFCLAS  computer  system  during 
Period  Y.  Include  purchase  price,  monthly  rental, 
and  other  operation  and  maintenance. 
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2.  Identify  average  cost  per  person  by  personnel 
category  for  all  laboratory  personnel  during 
Periods  X and  Y. 

3.  Identify  total  personnel  costs  for  all  laboratory 
personnel  for  Periods  X and  Y. 

4.  Identify  cost  per  copy  of  laboratory  clinical  forms 
by  kind  of  form  used  in  Periods  X and  Y. 

5.  Identify  cost  of  capital  expenditures  for  the 
clinical  laboratory  from  the  first  data  collection 
period  through  the  final  data  collection  period. 

6.  Identify  average  cost  per  person  by  personnel  cate- 
gory of  personnel  who  file  laboratory  results  in 
medical  records  in  the  Medical  Records  Department 
during  Periods  X and  Y. 

7.  Identify  average  cost  per  person  by  personnel  cate- 
gory of  personnel  who  file  laboratory  results  in 
medical  records  in  nursing  units  during  Periods  X 
and  Y . 

F.  Note  on  Sample  Size  for  Observation  and/or  Sampling  Studies 

In  order  to  conduct  the  Period  X evaluation,  an  estimate 
of  sample  size  is  required  for  the  following  areas: 

• Outpatient  Medical  Records  (OMR) 

• Inpatient  Medical  Records  (IMR) 

• Errors  in  arriving  outpatient  laboratory  slips 

• Duration  of  telephone  calls 

• Service  time  at  the  reception  desk 

• Total  patient  time  at  the  laboratory. 
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In  addition,  we  want  to  determine  the  effect  of  AFCLAS 
on  laboratory  slip  turnaround  time  and  on  frequency  of  tele- 
phone calls.  However,  if  all  slips  are  Xeroxed  during  the 
20-day  evaluation  period,  we  will  have  an  ample  number  from 
which  to  draw  a sample  after  Period  X in  order  to  assess 
turnaround  rime.  Also,  recording  of  all  telephone  calls  by 
the  receptionist  for  the  second  and  third  week  of  Period  X 
should  provide  a sufficient  number  with  which  to  analyze 
telephone  call  frequency. 

There  are  three  main  factors  influencing  the  sample  size. 
The  first  of  these  is  that  we  wish  to  test  hypotheses  on  the 
effect  of  AFCLAS  in  each  of  the  above  areas,  as  opposed  to 
merely  making  confidence  interval  statements  about  population 
parameters.  A second  and  related  consideration  is  that  we 
are  dealing  with  a two-sample  problem,  i.e.,  before  and  after 
AFCLAS,  in  contrast  to  hypotheses  on  parameters  of  a single 
population.  Finally,  we  shall  estimate  the  required  sample 
sizes  by  specifying  a probability  distribution  for  each  area. 

1.  Outpatient  Medical  Records  (OMR) 

This  will  be  a study  of  (a)  the  percentage  of  laboratory 
slips  not  filed  in  the  record  after  a month  and  (b)  the 
time  required  for  slips  to  reach  OMR.  If  p^  denotes  the 
percentage  of  slips  that  can’t  be  filed  before  AFCLAS  and  P2 
the  percentage  of  slips  that  can't  be  filed  after  AFCLAS,  then 
in  (a)  we  wish  to  test  the  null  hypothesis  p^  = against 

the  alternate  hypothesis  p^  # P2*  In  any  test  of  an 

hypothesis  we  must  specify  the  probability  a,  the  chance  that 
we  are  willing  to  take  that  we  have  a nonrepresentative  sample 
that  causes  us  to  reject  when  in  fact  it  is  true.  Clearly, 
we  would  like  m to  be  low.  We  must  also  specify  the  proba- 
bility 6 such  that  1-6,  the  chance  that  we  are  correctly 
rejecting  H^,  is  high.  Finally,  we  must  select  the  amount  of 
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change  6 between  and  p^,  which  we  would  like  to  be  able  to 
detect  by  testing  We  shall  regard  5 = |p^  - P2I  = .10  as 

a significant  change.  We  estimate  our  sample  sizes  similar  to 
Reference  1 based  on  a binomial  distribution,  since  the  labora- 
tory slips  belong  to  one  of  two  classes— either  they  are  filed 
in  the  record  or  they  are  not  filed.  The  sample  sizes  for 
Period  X (and  also  for  Period  Y)  are  given  by 

N 

where  2 and  2 6 are  the  a/2  and  6 points  of  the 
cumulative  standardized  normal  distribution  function,  respec- 
tively. Table  1 reflects  the  sample  sizes  needed  to  detect 
6 = .10  for  various  a and  1-6  values  where  p is  the  larger  of 
Pj^  and  P2.  Previous  studies  indicate  that  p^^  = .20  may  be  a 
minimum.  Sample  sizes  from  the  above  formula  were  checked 
using  Reference  2 and  where  minor  differences  were  found,  the 
larger  sample  sizes  were  used  for  Table  1.  Using  a = .10  and 
1-6  = .90  as  reasonable  probabilities,  we  project  a sample 
size  of  N = 400  laboratory  slips  or  25  slips  per  day  for  16 
days.  A means  by  which  to  select  random  samples  of  size  25 
will  be  provided  by  ANSER. 

The  second  part  of  the  OMR  study  deals  with  the  time 
required  for  slips  to  reach  the  record  room.  We  postulate  an 
exponential  distribution  for  this  time,  i.e., 

f (x)  = 0e 

so  that  the  mean  time  required  for  a slip  to  reach  records  is 

E (X)  = 1/6. 


100 


I 


M 

CQ 

< 

Eh 


101 


If  1/6^  and  1/02  denote  the  true  mean  times  before  and  after 
AFCLAS  respectively,  then  we  wish  to  test  the  hypothesis  that 
l/6j^  = 1/02  ~ ^ against 

assumption  of  exponentiality  is  a traditional  one  in  treating 
service  time  data  (Reference  3) . 

Let  and  X2  be  the  sample  mean  times  before  and  after 
AFCLAS,  each  based  on  a sample  of  size  N.  Then  it  can  be 
shown  that  (0,/0_)  x, /x_  follows  an  F_„  distribution.  If 

X 2 12  2N  f 2n 

4>  = 0j^/02,  our  sample  sizes  may  be  determined  from 


1-B  Prob  ^F2N,2N  ~ *^^0/2 , 2N,  2N^  ^^2N,2N  ^ ‘*’^l-a/2 , 2N,  2N^ 


where  a and  6 are  as  before,  and  F and  F,  „„ 

a/2,2N,2N  l-a/2,2N,2N 

represent  the  a/2  and  l-a/2  points  of  the  cumulative  F2j^  2n 
distribution,  respectively.  Table  2 gives  the  sample  sizes 
needed  to  achieve  1-6  = .90  for  various  a and  (J)  values. 


TABLE  2 


SAMPLE  SIZES  FOR  1-6  = .90 


95/100 

90/100 

(or  100/95) 

(or  100/90) 

.10 

960 

440 

.05 

1100 

480 

The  parameter  <|)  represents  a true  ratio  of  mean  service  times 
since 


<t>  = (l/02)/{l/0i)  . 

Thus,  if  we  would  like  to  be  able  to  detect  a change  from  100 
time-units  before  AFCLAS  to  90  time-units  after  AFCLAS  (or  90 
units  before  AFCLAS  to  100  units  after  AFCLAS)  with  a = .10, 
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1-6  = .90,  we  require  a sample  of  440  observations.  Due 
to  the  relative  ease  of  Xeroxing  the  slips  at  OMR,  it  was 
originally  felt  that  a sample  size  of  N=1000  would  be  de- 
sirable for  this  portion  of  the  study.  This  would  necessi- 
tate determining  the  age  of  these  slips  (in  days)  for  a 
period  of  1 week  based  on  an  average  of  200  slips  per  day 
arriving  at  the  record  room  for  filing.  However,  the 
analysis  may  require  stratification  of  the  slips  by  clinic, 
thereby  decreasing  the  sample  size  available  for  each  clinic. 
Therefore,  we  estimate  2 weeks  as  the  minimum  time  needed 
to  give  us  adequate  sample  sizes.  Sample  sizes  were  also 
calculated  based  on  a normal  distribution  of  service  times 
and  using  a small  sample  of  data  from  the  OMR  at  Wright- 
Patterson  AFB.  These  sample  sizes  were  found  to  be  substan- 
tially higher  than  those  of  Table  2 due  to  a large  variance 
in  arrival  times. 

2.  Inpatient  Medical  Records  (IMR) 

This  portion  of  the  study  will  also  be  treated  as  a 
binomial  distribution  since  a search  will  be  conducted  for 
the  laboratory  slips  in  the  record  on  the  wards  after  they 
have  left  the  laboratory.  The  slips  could  then  be  classified 
as  having  reached  the  record  in  the  ward  or  not  having 
reached  the  ward.  Also,  after  a month,  a search  of  IMR  will 
be  conducted  to  determine  how  many  of  the  slips  not  found 
in  the  record  on  the  wards  have  reached  the  record.  Again 
based  on  Table  1,  a single  sample  of  400  slips  or  25  slips  a 
day  for  16  days  can  be  used  for  the  ward  search,  followed  by 
a month-end  search  of  IMR  for  those  slips  not  found  on  the 
wards. 

3.  Errors  in  Arriving  Outpatient  Laboratory  Slips 

Once  more  we  will  require  the  observation  of  approxi- 
mately 400  slips  since  a slip  can  be  classified  as  either 


103 


being  in  error  or  not  being  in  error.  A slip  is  in  error  if 
one  or  more  of  the  following  exists: 

• No  patient  name 

• Illegible  patient  name 

• No  Social  Security  number 

• Illegible  Social  Security  number 

• Incomplete  Social  Security  number 

• No  clinic  or  ward  ncime 

• Illegible  clinic  or  ward  name, 

A slip  is  defined  to  be  correct  if  all  required  information 
is  on  the  slip  and  legible,  or  if  it  contains  one  or  more 
minor  errors  (all  possible  errors  except  the  seven  listed 
above) . Based  on  an  average  of  16  arrivals  per  hour 
(Reference  4),  we  would  require  approximately  25  hours  of 
observation.  We  feel  that  a sufficient  number  of  laboratory 
slips  will  be  checked  based  on  an  observation  time  of  20  hours 
or  two  1/2-hour  intervals  a day. 

4.  Duration  of  Telephone  Calls 

This  will  be  recorded  in  seconds.  From  Table  2,  based 
on  an  exponential  distribution,  we  must  have  at  least  400 
calls;  we  would  like  to  have  more.  At  an  average  of  10  calls 
an  hour  (Reference  4),  this  is  about  40  hours  of  phone  obser- 
vation or  2 hours  a day  in  eight  15-minute  observation  periods. 
Sample  sizes  calculated  from  an  assumed  normal  distribution 
and  using  data  from  Keesler  AFB  are  much  higher  estimates. 

5.  Service  Time  at  the  Reception  Desk  and  Total  Patient 
Time  at  the  Laboratory 

These  may  be  conducted  simultaneously  since  handing  out 
the  time  in/time  out  cards  to  the  patients  requires  very 
little  time.  Service  time  at  the  reception  desk  will  be 
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measured  in  minutes.  We  assume  an  exponential  distribution 
for  both.  If  we  require  about  450  observations  from  Table  2, 
then  based  on  an  average  of  about  15  observations  per  hour 
(Reference  4) , this  will  require  about  30  hours  of  observa- 
tion of  1.5  hours  per  day.  This  could  be  done  in  three  30- 
minute  intervals.  Again,  the  normal  distribution  using 
Keesler  AFB  data  for  service  time  at  the  reception  desk,  and 
using  data  from  Reference  2 for  total  patient  time  at  the 
laboratory,  yields  higher  sample  sizes  than  those  in  Table  2. 

ANSER  will  provide  a single  observation  schedule  for 
numbers  3 through  5. 

G.  Note  on  Choosing  a Random  Sample 

The  following  is  a method  for  choosing  a random  sample 
of  size  25  from  a population  of  size  n where  n = number  of 
Xeroxed  laboratory  slips  from  which  the  sample  is  to  be  drawn. 
As  a result  of  the  sequential  number  stamp,  each  slip  will 
have  a 3-digit  number  from  001  to  n. 

Pick  an  arbitrary  starting  point  in  the  Chemical  Rubber 
Company  (CRC)  Table  XII. 4 (Reference  5),  e.g.,  the  first  3- 
digit  number  group  on  page  480,  line  10,  column  (7) , which  is 
388.  Continue  down  this  column  observing  568,  186,  363,  etc. 
If  the  random  number  observed  is  from  001  to  n,  include  the 
slip  with  the  corresponding  number  in  the  saunple  of  size  25; 
otherwise  continue  to  the  next  random  number.  If  a number 
(slip)  has  already  been  included  in  the  seimple  and  is  observed 
again,  continue  to  the  next  random  number.  To  illustrate, 
the  first  slip  selected  under  the  above  sampling  scheme  for  a 
population  of  size  n = 200  would  be  slip  number  186.  At  the 
bottom  of  the  page,  proceed  to  the  next  3-digit  number  group 
in  line  1,  columns  (7)  and  (8),  which  is  791,  825,  792,  etc. 
Follow  this  procedure  until  a sample  of  size  25  has  been 
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collected.  Of  course,  a different  starting  point  should  be 
used  for  each  sample  taken.  The  same  process  can  be  applied 
for  selecting  samples  of  size  10  by  observing  2-digit  random 
numbers. 

H.  Observation  Schedule 

This  section  consists  of  Table  3,  the  Observation 
Schedule  for  Period  X and  Period  Y data  collection. 
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TABLE  3 

OBSERVATION  SCHEDULE 


e Observe  number  of  lab  slips  m error 

r • Observe  service  time  at  rerepfion  desk  and  band  out  time  in  time  out  cards 
t^  • Observe  call  durations  durmq  first  IS  minute  period 

t2  * Observe  call  durations  during  second  IS  minute  period 
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IV.  DATA  COLLECTION  TASKS  REQUIRING  THE  ASSISTANCE  OF 
LABORATORY  PERSONNEL  (PERIOD  X) 

Laboratory  reception  area  personnel  will  be  asked  to  do 
the  following: 

1.  Keep  a record  of  all  telephone  calls  received  during 
the  second  and  third  week  of  Period  X.  (A  data 
collection  form  will  be  provided.) 

2.  Collect  from  a sample  of  450  patients  the  form 
handed  out  by  MET  personnel  that  is  used  to  measure 
patient  time  in  the  laboratory. 

3.  Remind  hospital  staff  to  record  the  time  of  depar- 
ture from  laboratory  and  area  to  which  laboratory 
reports  will  be  delivered.  (A  data  collection  form 
will  be  provided.) 

4.  Record  time  of  departure  to  deliver  laboratory 
reports  to  the  wards  or  clinics  and  general  desti- 
nation. (A  data  collection  form  will  be  provided.) 

Other  laboratory  personnel  will  be  asked  to  do  the 
following : 

1.  Record  return  response  calls  for  inquiries  for  test 
status  or  test  result.  (A  data  collection  form  will 
be  provided.) 

2.  Record  the  time  leaving  and  the  time  of  return  to 
the  laboratory  from  all  specimen  collection  trips 
to  the  wards  or  clinics.  (A  data  collection  form 
will  be  provided.) 

3.  Remind  hospital  staff  to  record  the  time  of  departure 
from  the  laboratory  and  area  to  which  laboratory 
reports  will  be  delivered.  (A  data  collection  form 
will  be  provided.) 
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4.  Record  time  of  departure  to  deliver  laboratory 
reports  to  the  wards  or  clinics  and  general  desti- 
nation. (A  data  collection  form  will  be  provided.) 

5.  Record  workload  information  and  provide  it  to  MET 
personnel.  The  requested  workload  information 
includes  the  following; 

- The  total  daily  number  of  tests  performed  for 
each  kind  of  test  subdivided  into  inpatient  and 
outpatient  categories 

The  daily  number  of  stat  tests  performed  for  each 
kind  of  test  for  the  sections  of  Hematology, 
Urinalysis,  and  Stat  Laboratory 

- The  total  daily  number  of  request  slips  by  section 
subdivided  into  inpatient  and  outpatient  categories. 
(A  data  collection  form  will  be  provided.) 

- The  daily  number  of  stat  request  slips  for  the 
sections  of  Hematology,  Urinalysis,  and  Stat 
Laboratory  subdivided  into  inpatient  and  outpatient 
categories.  (A  data  collection  form  will  be 
provided. ) 

- Make  available  for  Xeroxing  historical  workload 
reports . 

6.  Record  the  name  of  personnel  on  duty  in  the  clinical 
laboratory  each  day  subdivided  by  laboratory  section. 

7.  Xerox  all  completed  laboratory  reports  when  ready 
for  distribution. 

8.  Complete  a laboratory  staff  survey  questionnaire 
(takes  approximately  15  minutes) . 

9.  Cooperate  in  answering  questions  and  providing  infor- 
mation about  procedures  to  MET  personnel. 
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V.  DATA  COLLECTION  TASKS  REQUIRING  THE  ASSISTANCE 
OF  PERSONNEL  OUTSIDE  THE  LABORATORY  (PERIOD  X) 

The  following  groups  of  people  outside  the  laboratory 
will  be  asked  to  complete  a questionnaire: 

• Physicians  (10-20  minutes) 

• Registered  nurses  (10  minutes) 

• Laboratory  patients  (10  minutes) . 

The  Registrar  will  be  asked  to  provide  the  following 
information : 

• Daily  admissions  and  discharges  (Admiss  ons  and 
Discharge  Report) 

• Daily  census  (Commander's  Report). 

The  Noncommissioned  Officer  in  Charge  of  each  outpatient 
clinic  will  be  asked  to  provide  the  daily  number  of  out- 
patient visits.  (A  data  collection  form  will  be  provided.) 

Personnel  from  outpatient  clinics  and  inpatient  nursing 
units  will  be  asked  to  sign  out  when  they  pick  up  completed 
laboratory  reports. 

Personnel  in  the  Outpatient  Medical  Records  Department 
will  be  asked  to  do  the  following: 

• Allow  a data  collector  to  Xerox  laboratory  reports 
ready  to  be  filed 

• Pull  medical  records  to  be  screened  by  data  collectors 
(approximately  500  records) 

• Aid  data  collectors  in  tracking  down  laboratory  reports 
not  yet  filed  in  the  outpatient  medical  record. 

Inpatient  Medical  Records  personnel  will  be  asked  to  pro- 
vide information  that  will  allow  estimation  of  the  fraction  of 
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inpatient  laboratory  reports  that  are  filed  in  inpatient 
medical  records  rather  than  on  the  wards.  (They  may  be 
asked  to  count  or  to  allow  a data  collector  to  count  labora- 
tory reports  by  the  date  the  report  was  ready  for  distri- 
bution. ) 

Hospital  staff  will  be  asked  to  aid  data  collectors 
observing : 

• Time  required  to  file  a laboratory  report  in  the 
outpatient  medical  record 

• Time  required  to  file  a laboratory  report  in  the 
inpatient  medical  record 

On  the  ward 

In  Inpatient  Medical  Records 

• Time  required  to  fill  out  a laboratory  request  slip 

In  the  clinics 

On  the  wards 

• Time  required  to  prepare  and  file  a laboratory  flow 
sheet  (including  the  number  and  percentage  of  patients 
requiring  flow  sheets) . 

The  Registrar/Hospital  Administrator  will  be  asked  to  pro- 
vide information  about  changes  that  occur  between  Period  X 
and  Period  Y. 
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VI.  DATA  COLLECTION  FORMS 


This  part  of  Appendix  B is  a compilation  of  all  data 
collection  forms  that  were  used  in  the  evaluation  of  AFCLAS 
at  MCWP. 
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List  of  Forms 


C 


Form  1:  Record  of  Telephone  Calls  to  Wright-Patterson 

Clinical  Laboratory  Reception  Area 

Form  2:  Followup  for  Calls  Made  from  Laboratory  Sections 

at  Wright-Patterson  AFB 

Form  3;  Record  of  Telephone  Calls  (to  be  Completed  by 
Laboratory  Personnel) 

Form  4;  Record  of  Telephone  Calls  to  Malcolm  Grow 
Clinical  Laboratory  Reception  Area 

Form  5:  Record  of  Telephone  Calls  (to  be  Completed  by 

Reception  Area  Personnel) 

Form  6:  Daily  Workload  Record 

Form  7;  Request  Slip  Record 

Form  8;  Stat  Request  Slip  Record 

Form  9:  Reception  Area  Observation  Record  (Errors  in 

Arriving-Outpatient  Request  Slips) 

Form  10;  Reception  Area  Observation  Record  (Service  Time 
at  the  Laboratory  Reception  Desk) 

’^orm  11:  Time  In/Time  Out  Card 

Form  12:  Record  of  Return  from  Specimen  Collection/ 

Departure  for  Result  Slip  Distribution 

Form  13;  Record  of  Tests  Sent  Out  to  (Received  from)  Other 
Clinical  Laboratories 

Form  14:  Report  of  Outpatient  Clinic  Visits 

Form  15:  Report  of  Inpatient  Census 

Form  16:  Turnaround  Time 

Form  17;  Inpatient  Reports 

Form  18;  Outpatient  Reports 
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RECORD  OF  TELEPHONE  CALLS  TO  WRIGHT  PATTERSON  CLINICAL  LABORATORY  RECEPTION  AREA 


Type  of  Caf*  Inqufry  fl).  Noninqu'ry  (NM.  Cad  o'lqinaimq  >n  lab  <0> 

Caiaqory  of  Ca((c<  WD,  Nu*w  <N).  Orhar  hotp>ta>  staff  fMS>.  Patient  <P).  Other  <0( 

Location  of  Cafiei  Spacific  ward  or  chnic.  outside  <0( 

O«v>oaiiion  conv‘eted  at  reception  area  (Cl.  <ecepi>omst  will  cad  back  (CB).  referred  to  lab  section  (identity  section) 
Received  By  Receptionist  IR).  Laboratory  Technician  (LT| 
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RECORD  OF  TELEPHONE  CALLS* 
(To  be  completed  by  Laboratory  Personnel) 


For  all  telephone  calls  placed  by  laboratory  personnel  m response  to  a previous  inquiry,  please  complete  the  section  entitled  "Record  ot  Inquiry  Response  Calls  " 

For  all  visits  for  information  on  test  status  or  test  results,  please  complete  the  section  entitled  "Visits  for  information  on  Test  Status  or  Test  Results." 

NOTtS 

Inquiry  Cali  Any  tel€p^one  request  (or  test  status  or  test  results  Location  Called  Name  o(  specific  ward  or  chnic  Cate9orv  ot  Person  Called  MU  Nutse  iNl.  Other  hospital  staff  (HSI 

ANSER  (March  1976)  F( 
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J 


RECORD  OF  TELEPHONE  CALLS 
(to  be  completed  by  Reception  Area  Personnel) 


J 

f 


1 


i 

I 

I 


I 
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ANSER  (M«r^  197S)  FORMS 


Routine 


Standard  Slips  by 
Laboratory  Section 


Paqe of Pages 


L22 


RECEPTION  AREA  OBSERVATION  RECORD 
(Errors  in  Arriving  Outpatisnt  Request  Slips) 


Pige of Pages 


Observation  Number  , 
Date 


Time  From. 


Patient's  , Clime/  Telephone  Rank 

Name  Ward  No  Relation 


I 


■ 


t Error  Millirtg  (M).  Mlogibl*  IIL).  Incomplett  <SSAN  only)  (1C) 


ANSCP  (March  1975) 
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Psg« of Pages 

RECEPTION  AREA  OBSERVATION  RECORD 
(Service  Time  at  the  Laboratory  Reception  Desk  I 

nhrjruaimn  Niimhei  Date Time  From Tn  Observer 


Patients 

Observed 

r— “ 
Timi  of 
Arrivil 

Service 
T ime 

Type  of 
Patient 

Instructions 

Given 

Comments 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

It 

12 

13 

14 

15 

16 

17 

18 

19 

20 

Notes 

Typ«  of  Pit««nt  In  in.  Out  (0) 

ln«truction«  Gtv«n  by  R*c«ption<ft  Yt«  lY).  No  INI 


ANSER  (March  1976) 
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TIME  IN 


Thf  time  that  patients  spend  in  the 
clinical  lahoratory  is  being  studied  and 
your  cooperation  is  needed  Please 
return  this  card  to  the  receptionist  or 
the  person  who  gave  it  to  you  when  you 
leave  the  laboratory. 

Are  you  here  tor  a (ilucose  Test'.’  a Yes 

D No 


ANSEn  (March  1975) 


FORM  11 


TIME  OUT 


P«g« of PagBS 


RECORD  OF 


RETURN  FROM  SPECIMEN  COLLECTION 
DEPARTURE  FOR  RESULT  SLIP  DISTRIBUTION 


Location . 


. Date  . 


. Certified  by  . 


’SpKiman  colltction  teaint  antir  collection  location,  result  distribution  personnel  enter  destination  of  laboratory 
reports  to  be  delivered. 


ANSER  iM»cA  tgyei  form  12 
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ANSER  <M«rc^  197SI  fORM  U 


REPORT  OF  INPATIENT  CENSUS 
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TURNAROUND  TIME 


I hit  form  will  only  be  used  it  laboralory  reports  cannot  be  Xeroxed. 


INPATIENT  REPORTS 


ANSER  (M«rch  19761  FORMW 


OUTPATIENT  REPORTS 


! 


o 


□lie  Slip 
found 
in  Record 

1 

1 

1 

1 

1 

1 

1 

— 

— 

— 

Record  Search 

Comments 

1 

1 

1 

1 

1 

1 

1 

r 

r 

Record 

Found 

1 

1 

1 

1 

1 

1 

1 

Dale  of 
Search 

1 

1 

1 

1 

1 

1 

1 

Location 

1 

1 

1 

1 

1 

1 

1 

Record  Search 

Comments 

1 

1 

1 

1 

1 

1 

1 

Record 

Found 

1 

1 

1 

1 

1 

1 

1 

Date  of 
Search 

1 

1 

1 

1 

1 

1 

1 

Localion 

1 

1 

1 

1 

1 

1 

1 

^ t 

If 

1 

1 

1 

1 

1 

1 

1 

< 

CO 

1 

1 

1 

1 

1 

1 

1 

•» 

E 

m 

X 
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APPENDIX  C 
SURVEY  SCHEDULES 

1 


i 
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NOTE  ON  APPENDIX  C 


The  schedules  in  this  appendix  are  designed  to  measure 
the  attitudes  toward  the  clinical  laboratory  and  acceptance 
of  AFCLAS.  The  schedules  were  designed  not  to  prove  some 
expected  dramatic  changes,  but  rather  to  measure  change,  if 
in  fact  there  is  change. 
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I.  AFCLAS  AND  THE  PERSONNEL  SYSTEM  OF  THE  HOSPITAL 

The  effectiveness  of  a technological  change  such  as 
AFCLAS  should  be  measured  by  more  factors  than  its  conformity 
to  design  specifications.  AFCLAS  will  impinge  upon  the  person- 
nel system  of  the  laboratory  and  other  personnel  systems  of 
the  hospital.  Its  acceptance  by  hospital  personnel  will 
determine  how  well  technical  capability  will  be  converted  into 
operational  capability.  Although  the  relationship  between 
patient  satisfaction  and  the  effectiveness  of  AFCLAS  is  not 
as  evident  as  the  relationship  of  hospital  personnel  to  AFCLAS, 
it  is  reasonable  to  assume  that  some  relationship  exists. 

AFCLAS  will  impinge  upon  personnel  subsystems  within  the 
hospital  personnel  system  in  different  degrees.  It  seems 
reasonable  to  expect  that  the  perceived  impact  of  AFCLAS  will 
be  a function  of  the  degree  to  which  AFCLAS  impinges  upon  a 
person's  daily  tasks  and  the  perceived  significance  of  these 
tasks.  AFCLAS  will  have  the  greatest  impact  on  laboratory 
personnel  because  their  tasks  are  more  directly  related  to  it 
than  are  any  other  personnel  categories.  Other  personnel 
subsystems  in  the  hospital  will  be  less  affected  since  AFCLAS 
will  have  impact  on  only  some  of  their  tasks.  Physicians, 
for  example,  will  devote  a very  small  part  of  their  time  to 
consideration  of  laboratory  reports.  The  clinical  laboratory 
report  is  only  one  of  his  many  diagnostic  tools — albeit  an 
important  oneo 

Studies  of  job  satisfaction  generally  do  not  show  a 
significant  correlation  between  job  satisfaction  and  productivity. 
Job  satisfaction  studies  do  show  a positive  correlation  between 
job  satisfaction  and  personnel  retention  and  a negative  corre- 
lation between  job  satisfaction  and  absenteeism. 

AFCLAS  is  a man-machine  system  and  it  will  not  attain  its 
expected  technical  capability  unless  it  is  accepted  by  the 
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users.  Industrial  sociologists  and  psychologists  accept  as 
axiomatic  widespread  resistance  to  technological  innovation. 

The  prejudice  associated  with  this  resistance  to  innovation 
is  confirmed  by  the  initial  disruption  caused  by  a new  system. 
Acceptance  by  employees,  therefore,  cannot  be  expected  until 
the  system  has  been  operating  smoothly  for  some  time.  Thus, 
the  Period  Y schedules  should  be  administered  as  late  as 
possible  after  the  system  is  operational. 

A Job  Satisfaction  Schedule  will  measure  acceptance  by 
laboratory  personnel.  Other  schedules  will  measure  acceptance 
by  physicians  and  registered  nurses.  In  addition,  patient 
satisfaction  with  the  clinical  laboratory  will  be  measured 
before  and  after  implementation  of  AFCLAS. 

The  terms  "schedule"  and  "questionnaire"  are  often  used 
interchangeably  in  studies  of  data  collection  methods. 

Schedule,  however,  is  a more  general  term  and  includes  all 
categories  of  data  to  be  collected  by  the  researcher  by 
systematic  observation.  Schedules  may  range  from  the  categories 
of  behavior  to  be  observed  and  recorded  by  the  participant 
observer  of  group  behavior  to  forced  choice  answers  to 
questions  on  a form  to  be  filled  out  by  the  respondents. 

Questionnaires  are  standardized  schedules  in  which  each 
question  is  proposed  to  each  respondent  in  the  same  way. 
Questionnaires  may  be  used  in  interviews  or  may  be  in  a form 
that  the  respondent  may  read  and  reply  to  in  writing. 

A scale  is  a method  of  measuring  variables  in  a social 
environment  based  on  response  to  more  than  one  question  by 
respondents.  Scales  may  be  developed  in  which  the  response 
to  a given  question  is  not  significant  in  itself  but  is 
significant  as  part  of  a composite  score;  e.g.,  the  Laboratory 
Patient  Scale.  The  physician  and  nurse  questionnaires  are 
questionnaires  in  the  instruction  section  and  questionnaires 
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that  form  scales  in  the  remainder.  Each  question  provides 
data  that  is  significant  per  se.  Each  question  also  is  part 
of  a scale  that  will  provide  a composite  score  for  each 
respondent. 
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II 


DESCRIPTION  OF  SCHEDULES  AND  INTERVIEWS 


This  section  describes  generally  some  of  the  techniques 
and  considerations  inherent  in  the  development  of  the  schedules 
and  interviews. 

A.  Physician  and  Registered  Nursing  Staff  Schedules 

The  physician  and  registered  nurse  schedules  are  designed 
to  measure  the  perception  of  AFCLAS  and  of  the  clinical 
laboratory  before  and  after  the  installation  of  AFCLAS. 

Rigorous  measures  of  the  validity  of  the  instrument  are  impos- 
sible because  the  individuals  administering  the  schedule  have 
no  souLce  of  information  on  the  true  perception  of  AFCLAS 
(if  it  were  known  there  would  be  no  need  for  the  instrument) . 

The  validity  of  an  instrument  may  be  defined  as  the 
extent  to  which  differences  in  score  reflect  true  differences 
between  perceptions  of  AFCLAS  or  the  clinical  laboratories 
before  and  after  installation  of  AFCLAS. 

Since  there  is  no  known  measure  of  true  perception, 
validity  must  be  determined  by  other  measures.  Prognostic 
validity  (i.e.,  the  accuracy  of  predictions  based  on  the 
schedule)  and  construct  validity  {i.e.,  the  degree  to  which 
an  individual  possesses  some  characteristic  reflected  in  test 
performance  and  the  scores  on  the  test) , which  correlate  with 
performance  and  behavior  cannot  be  used  to  measure  the  validity 
of  the  instruments  below. 

The  measure  of  validity  of  physician  and  registered 
nursing  staff  schedules  is  the  self-evident  relevance  of  the 
questions  on  the  schedule  to  what  is  to  be  measured.  "^his 
is  referred  to  in  literature  on  survey  research  as  face 
validity — ^valid  "on  the  face  of  it." 
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The  question  of  validity  of  instruments  is  an  unsolved 
problem  in  social  research  and  a solution  is  not  in  sight. 
Although  the  social  science  researcher  cannot  ignore  the 
problem,  he  can  minimize  it  by  avoiding  the  pitfalls  of 
questionnaire  construction.  If  the  schedule  he  designs  is 
valid,  it  will  not  be  rejected  for  other  reasons.  In  other 
words,  the  researcher  should  avoid  all  known  practices  that 
degrade  the  validity  of  schedules. 

B.  Laboratory  Staff  Questionnaire 

Employee  satisfaction  has  been  studied  by  psychologists, 
social  psychologists,  sociologists,  and  management  scientists 
for  a long  time.  The  variables  identified  are  extensive. 
However,  two  general  categories  of  variables — hygiene  (i.e., 
the  work  environment)  and  motivation — are  widely  accepted. 
Within  these  two  major  categories,  four  classes  of  variables 
are  generally  accepted.  These  are:  variety,  autonomy,  task 

identity,  and  feedback.  Two  additional  variables— dealing 
with  others  and  friendship  opportunities — have  sometimes  been 
used  in  employee  satisfaction  research.  In  the  laboratory 
staff  questionnaire,  each  question  was  written  to  investigate 
one  of  the  six  variables. 

C.  Laboratory  Patient  Satisfaction  Scale 

The  laboratory  patient  satisfaction  scale  is  designed  to 
measure  the  patient's  perception  of  the  clinical  laboratory 
and  is  based  on  the  theory  of  organizational  climate.  The 
basic  concept  is  that  the  behavior  of  the  membership  of  an 
organization  reflects  the  work  climate  of  the  organization. 

The  work  climate,  in  turn,  is  perceived  by  the  consumers  on 
the  basis  of  behavior  of  members  of  the  organization.  Finally, 
behavior  of  consumers  is  correlated  with  their  perception  of 
organization  climate.  In  other  words,  employee  and  consumer 
satisfaction  are  correlated.  For  purposes  of  this  study. 


142 


employee  satisfaction  is  an  independent  variable,  and 
consumer  satisfaction  a dependent  variable. 

D.  Medical  Records  Department  and  Admissions  and  Dispositions 
Department  Staff  Interviews  (Period  Y Only) 

Both  the  Medical  Records  Department  staff  and  the  Admis- 
sions and  Dispositions  Department  staff  are  small,  and  members  of 
these  staffs  appear  to  have  little  direct  contact  with  the 
laboratory.  A controlled  nondirective  interview  is  feasible  to 
provide  an  estimate  of  the  influence  of  AFCLAS  on  the  staff  of 
these  departments.  The  use  of  the  controlled  nondirective 
interview  at  the  initial  installation  sites  also  may  provide 
insight  that  will  be  useful  in  evaluation  of  future  installations. 


The  controlled  nondirective  interview  is  an  adaptation 
of  the  technique  developed  by  Carl  Rogers  for  counseling  and 
psychotherapy  to  interviewing.  The  interviewer  sets  the  stage 
by  describing  the  purpose  of  the  interview,  but  from  that  point 
on  the  respondent  communicates  in  his  own  way.  The  interviewer 
interjects  neutral  comments  to  keep  the  interview  going  and 
may  bring  the  interviewee  back  on  course  by  a directive  remark 
but  the  interviewer  is  essentially  respondent-centered. 

The  technique  for  interviewing  the  medical  records  staff 
might  begin,  after  introductions,  with  a statement: 


Interviewer 

Respondent 

Interviewer- 

The  interviewer 


Part  of  your  job  is  to  file  laboratory 
reports. 

Yes. 

You  place  the  laboratory  reports  in 
the  files? 

keeps  the  interview  going  by  statements 


such  as: 


I see . 
Hmm. 
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How  do  you  mean  that? 

- I don’t  quite  understand  that. 

- Why  do  you  think  it  is  that  way? 

Can  you  tell  me  more  about  that? 

- Is  there  anything  else? 

E,  thysician  Interviews  (Period  Y Only) 

During  Period  Y a random  sample  of  25  physicians  will  be 
interviewed  to  obtain  further  insight  into  the  physicians' 
perceptions  of  AFCLAS. 

The  technique  of  controlled  nondirective  interviewing 
will  be  used  to  obtain  responses  to  the  following  questions: 

• What  has  been  the  impact  of  AFCLAS  on  your  day-to- 
day  practice  in  the  delivery  of  health  care? 

• What  has  improved  since  the  introduction  of  AFCLAS? 

• What  has  deteriorated  since  the  introduction  of 
AFCLAS? 

• What  was  your  initial  reaction  to  AFCLAS,  and  how 
has  it  evolved  to  the  present? 

• What  improvements  in  AFCLAS  would  help  you? 
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III. 


SCALING  OF  SCHEDULES 


This  section  describes  how  the  physician,  registered 
nurse,  laboratory  staff,  and  patient  schedules  are  being 
scaled.  The  section  also  summarizes  the  method  for  inter- 
preting the  interviews  of  physicians.  Medical  Records  Depart- 
ment, and  Admissions  and  Dispositions  Department  staff. 

A.  Scaling  the  Physician's  and  Registered  Nurse's  Questionnaires 

The  AFCLAS  evaluation  plan  includes  questionnaires  for 
physicians  and  registered  nurses.  These  questionnaires  were 
developed  to  measure  the  attitudes  of  physicians  and  regis- 
tered nurses  toward  the  clinical  laboratory  before  and  after 
the  installation  of  AFCLAS.  The  physician's  questionnaire 
contains  two  parts,  and  the  registered  nurse's  questionnaire 
contains  one  part;  all  three  parts  will  be  scored  using  the 
scime  method.  The  respondent  is  asked  in  each  item  of  the 
questionnaire  to  express  an  opinion  by  checking  one  of  five 
phrases  or  one  of  six  quantities  that  relate  to  the  statement. 

The  measurement  of  attitude  change  from  before  to  after 
AFCLAS  would  be  simple  if  the  scores  on  individual  questions 
could  be  aggregated  and  a mean  obtained  for  the  attitude  of  a 
population  before  and  after.  This  cannot  be  done  by  treating 
each  question  as  an  equal  measure  of  attitude  toward  the 
clinical  laboratory,  since  a pretest  of  the  questionnaires 
indicated  the  respondents  attached  a wide  range  of  importance 
to  the  questions.  It  is  necessary,  therefore,  to  weight  the 
responses  to  the  questions  (which  has  the  effect  of  weighting 
the  questions)  in  order  to  obtain  a valid  measure  of  attitude 
change.  (See  Reference  1.)  The  general  method  for  scoring  one 
question  (repeated  for  all  questions)  is  outlined  as  follows 
and  then  described  in  more  detail; 
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1.  Compute  standard  score  for  each  response  (a 

question  has  either  5 or  6 possible  responses 
so  i = 1,  6)  from  the  Period  X data. 

2.  Determine  frequency  for  each  response  from  Period 
Y data. 

3.  Compute  a mean  standard  score  using  the  standard 
scores  from  1 above  and  the  Period  Y fre- 
quencies from  2 above. 

4.  Determine  if  the  change  in  acceptance  from  Period  X 
to  Period  Y is  statistically  significant  by  using 

a statistical  test  that  compares  the  mean  standard 
score  in  3 with  the  mean  standard  score  computed 
using  Period  X data  only. 

It  is  assumed  for  purposes  of  scaling  that  the  attitude 
toward  the  clinical  laboratory,  as  measured  by  each  question, 
is  normally  distributed.  For  each  question  for  the  Period  X 
data,  the  standard  deviation  from  the  estimated  mean  of  a 
normal  distribution  is  first  calculated.  Each  standard 
deviation  is  then  converted  to  a standard  score  with  a mean  of 
50  and  a standard  deviation  of  10. 

Computation  of  the  standard  deviations  and  the  standard 
scores  for  one  of  the  questions  in  the  physician's  question- 
naire is  described  below.  Ques»,ion  number  23  of  Physician's 
Questionnaire  i2  reads  as  follows 

AFCLAS  will; 

□ significantly  decrease 

□ slightly  decrease 

□ have  no  effect  on 

Q slightly  increase 

□ significantly  increase 

o no  opinion 


the  number  of  repeat 
requests  for  laboratory 
work. 
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Table  1 lists  the  data  for  the  above  question  obtained 
from  MCWP  during  Period  X. 


TABLE  1 

DATA  FOR  QUESTION  NO.  23  OF  PHYSICIAN'S 
QUESTIONNAIRE  #2  (PERIOD  X) 


Response 

Number 

Responses 

Number 

Checking 

Percent 

Checking 

Cumulative 

Percent 

Checking 

cr-Values 

Standard 

Scores 

1. 

significantly  decrease 

18 

31 

-1.14 

38.6 

2. 

slightly  decrease 

29 

50 

69 

0.16 

51.6 

3. 

have  no  effect  on 

10 

17 

19 

1.31 

63.1 

4. 

slightly  increase 

1 

2 

2 

2.44 

74.4 

5. 

significantly  increase 

0 

0 

0 

3.00 

80.0 

Number  of  Valid  Responses: 

58 

No  Opinio>^; 

28 

Blanks : 

3 

In  the  above  table,  the  percentages  have  been  rounded  to 
the  nearest  whole  number.  The  a-values  have  been  obtained  in 
the  following  way:  since  no  physicians  responded  to  number  5, 

"significantly  increase,"  the  mean  a-distance,  from  the  mean 
of  a normal  distribution,  was  set  to  3.00.  The  mean  a- 
distance,  from  the  mean  of  a normal  distribution,  of  the  2- 
percent  of  a normal  distribution  at  the  extreme  right  of 
the  distribution  is  obtained  by  going  to  Table  2,  Mean  a- 
Distances  from  the  Mean  of  Various  Percents  of  a Normal 
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TABLE  2 

MEAN  <7  DISTANCES  FROM  THE  MEAN  OF  VARIOUS 
PERCENTS  OF  A NORMAL  DISTRIBUTION 
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Source  Reproduced  trom  Garrett.  Ref,  2,  by  permission  ot  the  publisher 


Distribution.  In  the  table,  the  average  o-distance  from  the 
the  mean  for  2 percent  is  2.44  (column  0,  row  2).  Thus  a 
response  of  "slightly  increase"  is  given  an  a-value  of  2.44. 
Reference  to  Figure  1 will  make  this  clearer.  The  a-value 
for  "have  no  effect  on"  is  obtained  by  going  to  the  column 
headed  2 and  taking  the  entry  at  row  17,  which  is  1.31. 

This  means  that  when  2 percent  of  the  distribution  has  been 
used,  the  mean  distance  of  the  next  17  percent  from  the 
mean  of  the  normal  distribution  is  1.31.  The  percent  of 
respondents  checking  "slightly  decrease"  is  on  both  sides 
of  the  mean  of  the  distribution  (31  percent  to  the  right  of 
the  mean  and  19  percent  to  the  left) . Since  19  percent  of 
the  distribution  to  the  right  of  the  mean  has  been  accounted 
for,  31  percent  of  those  who  checked  "slightly  decrease"  will 
be  to  the  right  and  19  percent  to  the  left  of  the  mean  of 
the  distribution.  From  the  column  headed  19  and  the  row 
marked  31  by  Table  2,  the  mean  a-distance  for  the  31  percent 
is  0.41,  which  is  the  mean  o-distance  for  the  31  percent  to 
the  right.  From  column  31  (50  percent  - 19  percent)  and  row 
19,  the  entry  0.24  is  taken.  This  is  the  mean  a-distance  of 
the  19  percent  for  the  response  "slightly  decrease,"  which 
lies  to  the  left  of  the  mean.  The  overall  a-distance  of  the 
50  percent  responding  "have  no  effect  on"  is  then  computed 
as 


(0.31)  (0.41)  - (0.19)  (0.24) 

(TTTC 


Going  to  the  other  end  of  the  distribution,  31  percent 
responded  "significantly  decrease."  From  column  0 and  row  31, 
the  entry  is  1.14.  (This  becomes  -1.14  since  it  is  to  the 
left  of  the  mean  of  the  distribution.)  This  is  the  mean  a- 
distance  from  the  normal  distribution  mean  for  the  "significantly 
decrease"  response. 
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FIGURE  1 

SCALING  OF  THE  FIVE  RESPONSES  TO  QUESTION  23, 
PHYSICIAN'S  QUESTIONNAIRE  #2 


The  corresponding  standard  score  for  each  a-value  is 
obtained  by  moving  the  decimal  one  place  to  the  right  and 
adding  50.  This  is  to  convert  from  a [0,  1]  normal  distri- 
bution (mean  = 0,  a = 1)  to  a [50,  10]  normal  distribution 
(mean  = 50,  a = 10)  . 

The  average  standard  score  for  the  population  for 

q ’stion  number  23  as  computed  from  the  Period  X data  should 

equal  50,  and  any  deviation  is  due  to  roundoff  error: 

^ _ (18)  (38.6)  + (29)  (51.6)  + (10)  (63.1)  + (1)  (74.4)  + (0)  (80.0) 

^ 

= 49.9. 

Table  3 lists  the  data  for  question  number  23  obtained 
from  MCWP  during  Period  Y„ 


TABLE  3 

DATA  FOR  QUESTION  NO.  23  OF  THE 
PHYSICIAN'S  QUESTIONNAIRE  #2  (PERIOD  Y) 


Response 

Number 

Responses 

Number 

Checking 

1 

significantly  decrease 

2 

2 

slightly  decrease 

16 

3 

have  no  effect  on 

37 

slightly  increase 

26 

5 

significantly  increase 

21 

Number  of  Valid  Responses: 

102 

No  Opinion: 

9 

Blan)cs 

1 
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A mean  standard  score  for  Period  Y using  the  frequency 
of  responses  from  Period  Y and  the  standard  scores  from 
Period  X is  computed  as  follows; 

- _ (2)  (38.6)  + (16)  (51.6)  + (37)  (63.1)  + {26)  (74.4)  + (21)  (80.0) 

Xy  ^ 

= 67.2 


When  computing  a-values  and  standard  scores,  several 
special  cases  are  handled  as  follows: 

• If  there  are  no  responses  to  number  5,  then 
set  0 = +3.00,  which  implies  a standard  score 
of  80. 

• If  there  are  no  responses  to  number  1,  then  set 
a = -3.00,  which  implies  a standard  score  of  20. 

• If  there  are  no  responses  to  number  4 or  number 
5,  then  set  a = +3.00  for  both  responses,  which 
implies  a standard  score  of  80  for  both  responses. 

• If  there  are  no  responses  to  number  1 or  number  2, 
then  set  o = -3o00  for  both  responses,  which 
implies  a standard  score  of  20  for  both  responses. 

• If  a zero  response  occurs  between  two  non-zero 
responses,  use  the  following  formula  to  obtain 
the  standard  score  for  the  zero-response  item: 


standard  score  = lOX  + 50 

c 

where  X is  such  that 
c 


OO 


1 


\'  2 IT 


-X 


2 


/2 


e 


dx  = percent  checking  all 
previous  responses. 
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m 


The  statistical  test  to  determine  if  the  difference 
between  the  mean  standard  score  from  Period  X is  significant- 
ly different  from  the  mean  standard  score  from  Period  Y is 
described  below. 

Let 


X = mean  standard  score  computed  using 
Period  X data  only. 


f . 
1 


s 


mean  standard  score  computed  using  the 
frequencies  from  Period  Y and  the 
standard  scores  from  Period  X, 

the  number  answering  each  response  in 
Period  Y, 

an  estimate  of  standard  deviation, 


n = the  number  of  possible  responses  to 
a question  (n  = 5 or  6)  • 


To  test  for  statistical  significance,  compare  t with  tables 

of  Student’s  t statistic  using  t,  “(  S ^ i 

1^=1  i 
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B.  Scaling  the  Laboratory  Staff  Questionnaire 

The  Laboratory  Staff  Questionnaire  is  made  up  of  two 
parts.  The  first  part  is  a conventional  job  satisfaction 
scale.  The  second  part  is  a questionnaire  designed  solely 
to  obtain  data  on:  the  laboratory  environment;  the  individ- 

ual respondent's  activities  that  impinge  upon  the  time  he 
spends  in  the  laboratory,  e.g.,  time  worked  in  the  laboratory, 
overtime  work  in  the  laboratory,  military  duties  outside  the 
laboratory;  the  individual's  qualitative  evaluation  of  task 
frequencies  in  the  laboratory,  e.g.,  frequency  with  which  he 
is  rushed  in  performing  tasks;  and  the  individual's  evaluation 
of  the  quality  of  the  laboratory,  e.g.,  quality  of  reports, 
quality  control.  The  second  part  is  not  scaled.  The  first 
part  is  a Likert-type  scale. 

The  Likert-type  scale  is  an  ordinal  scale  of  attitudes 
toward  some  object  or  objects  in  the  environment — in  this 
case  attitude  toward  the  job  or  job  satisfaction. 

The  Laboratory  Staff  scale  for  the  first  part  of  the 
Laboratory  Staff  Questionnaire  is  a typical  Likert-type  scale 
with  five  response  categories.  Since  the  statements  all 
relate  to  a positive  attitude  toward  the  job,  each  item  can 
be  scored  from  1 to  5 beginning  with  "strongly  agree."  Any 
other  method  of  scoring  that  maintains  the  ordinality  of 
response  can  be  used.  Each  individual's  composite  score  is 
obtained  by  adding  his  item  scores.  The  second  part  of  the 
Laboratory  Staff  Questionnaire  will  not  be  scaled,  but  the 
responses  will  be  tallied  to  report  the  data  collected. 

C.  Scaling  the  Laboratory  Patient  Satisfaction  Scale 

The  Laboratory  Patient  Satisfaction  Scale  is  a Likert- 
type  scale  designed  to  measure  patient's  perception  of  the 
organizational  climate  of  the  clinical  laboratory. 
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The  first  15  questions  make  up  the  scale.  The  last 
question  divides  the  patients  into  two  categories — inpatient 
and  others. 

The  statements  that  make  up  the  scale  all  indicate  a 
positive  perception  of  the  laboratory  climate.  These 
responses  may  be  scored  from  1 to  5 beginning  with  "strongly 
agree."  A composite  score  for  individuals  is  obtained  by 
adding  the  responses  to  each  statement. 

D.  Interpreting  Interviews  With  Physicians  and  the  Staffs 
of  the  Medical  Records  Department  and  Admissions  and 
Dispositions  Department 

The  Medical  Records  Staff  interviews  and  the  Admissions 
and  Dispositions  Department  Staff  interviews  are  designed  to 
obtain  qualitative  data  on  tne  influence  of  AFCLAS  on  the 
respective  departments. 

The  interviews  will  be  the  controlled  nondirective  type. 
Although  the  number  of  respondents  will  be  small,  a large 
number  of  comments  can  be  expected.  The  analysis  of  the  data 
collected  can  be  simplified  by  categorizing  the  comments. 

The  categories  selected  must  be  broad  enough  to  include 
a large  number  of  comments  but  not  so  broad  as  to  include 
commients  that  are  obviously  dissimi''ar.  The  usual  approach 
is  to  provide  rather  broad  categories  and  break  each  category 
into  subcategories.  For  example,  responses  to  the  medical 
records  interview  might  be  placed  in  such  categories  as  filing 
and  retrieval.  Subcategories  under  filing  might  include: 
time,  readability,  quantity.  Subcategories  under  retrieval 
might  include:  lost  files,  time  to  retrieve. 

Comments  by  respondents  will  be  placed  in  categories,  and 
the  number  of  comments  in  each  category  will  be  determined. 

This  number  then  may  be  converted  into  a percent  of  respondents 
who  made  each  category  of  response,, 
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It  should  be  emphasized  that  the  results  of  analysis  are 
descriptive.  A descriptive  study  is  an  important  source  of 
hypothesis,  which  may  be  tested  scientifically  in  a follow-on 
study. 


156 


REFERENCES 


(1)  Henry  E.  Garrett.  Statistics  in  Psychology  and 

Education.  6th  ed.  New  York:  David  McKay  Company, 

Inc.,  1966;  .323-327  . 

(2)  Henry  E.  Garrett.  Statistics  in  Psychology  and 

Education : 68-69. 


157 


IV.  SURVEY  QUESTIONNAIRES 


The  survey  questionnaires  contained  in  this  section  are 
the  Period  Y version  administered  at  USAF  Medical  Center 
Wr ight-Patter son  AFB.  The  questionnaires  which  will  be 
administered  at  the  Malcolm  Grow  Medical  Center,  Andrews  AFB, 
have  the  name  of  the  hospital  changed,  but  otherwise  will  be 
the  same  questionnaires.  The  Period  X version  of  the  question- 
naires were  the  same  as  the  enclosed  questionnaires  but  with 
the  appropriate  verbs  changed  to  future  tense. 


Not 
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PHYSICIAN'S  gUESTIONNAIRE  #1 


Instructions 

I 1.  Consists  the  first  questionnsire  bssed  upon  your  knowledge  and  personal  experience  with  the  clinical 
I laboratories  sc  the  Wriqht-Patterson  Medical  Center  and  other  clinical  laboratories  (Air  Force  and  non-* 

Air  force) . 

I 2.  Then  complete  the  second  attached  questionnaire  entitled  "Physician’s  Questionnaire  «2 — AFCLAS." 

3.  Finally,  insert  the  ccxnpleted  questionnaire  into  the  accompanying  envelope,  seal  the  envelope,  and 
print  the  following  information  on  the  front  of  the  envelope-. 

Name 

Department . 

This  information  will  be  used  only  to  ensure  that  responses  from  all  physicians  at  Wright-Patterson 
Medical  Center  will  be  included  in  the  study.  The  sealed  envelope  will  be  opened  only  by  the  research 
team  of  Analytic  Services  Inc.  (ANSER) , and  the  information  you  provide  will  be  available  only  to  the  team.. 

1.  Laboratory  reports  at  this  hospital  are 


I 


much  more 
5or'ew^.at  more 
about  as 
somewhat  less 
much  less 


legible  tr.an  (as)  clinical  Itjoratory  reports  elsewhere. 


At  the  Wright-Patterson  Medical  Center  I have 

/ confidence  :n  the  clinical  laboratory  information  I use 
with  my  patients  than  (as)  I have  ha i at  other  hospitals. 


much  more 
more 

about  as  much 
less 

much  less 


Coverall,  with  respect  to  my  experience  with  the  clinical  laboratory  at  the  Wright-Patterson  Medical 
Center,  I am 

completely  satisfied, 
well  satisfied, 
reasonably  satisfied, 
somewhat  dissatisfied, 
completely  dissatisfied. 

At  the  Wright-Patterson  Medical  Center,  the  response  time  for  routine  laboratory  lesults  is 

I 


much  shorter  than 
somewhat  shorter  than 
about  as  short  as 
slightly  longer  than 
far  longer  than 


completely  confident 
very  confident 
reasonably  confident 
not  very  confident 
not  at  all  confident 


I would  expect  at  clinical  laboratories  generally. 


in  the  information  1 receive  from  the  clinical  laboratoryj 
at  this  hospital . 


6.  At  the  Wright-Patterson  Medical  Center,  the  resp'onse  time  for  routine  laboratory  results  is 

very  short, 
short . 
moderate . 
long . 

very  long. 

ANSER 

March  W-P  1976 
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At  this  hosj'itaX,  lahoratory  reports  are  lost 

much  less  often  than 

less  often  than  # 

about  as  often  as 
more  often  than  i 

much  more  often  than  ' 


I would  expect  in  this  kind  of  hospital  (Air  Force 
and  other) . 


At  the  Wriqht-ratterson  Medical  Center,  filing  of  clinical  laboratory  reports  in  patients’  charts  is 

100-percent  complete, 
very  complete, 
reasonably  complete, 
somewhat  incomplete . 
very  incomplete. 


At  the  Wr iqht-Patterson  Medical  Center,  it  is  necessary  to  rejeat  requests  for  laboratory  wrrk 


much  less  often  than  . 

somewhat  less  often  than  / 
about  as  often  as 
somewhat  more  often  than  \ 
much  more  often  than  ' 


would  exp^ect  for  clinical  labcjratcr les  of  this  kind 


(Air  Force  and  other) . 


During  the  last  week,  I called  the  clinical  laboratory 


none 

I- 5  times 
6-10  tines 

II- 15  times 
16-20  times 
over  20  times 


to  inquire  about  test  results. 


Curing  the  last  week. 


I- 5 
e-10 

II- 15 
16-20 
over  2C 


of  my  clinical  laboratory  reports  were  lost. 


During  tne  last  week,  I visited  the  clinical  laboratory  to  - heck  on  lab'’ ratory  test  status  or 
test  results  on 

no  j 

I- 5  / 

6-10  occasions. 

II- 15  . 

lf-20  \ 

over  20 

The  number  of  errors  in  reports  from  the  clinical  laboratory  at  this  hospital  is 

much  smaller  than  • 

somewhat  smaller  than  f 

about  the  same  as  I would  expect  for  clinical  laboratories  of  this  kind 

somewhat  larger  than  ^ (Air  Force  and  other), 

much  larger  than 

Dc  you  ever  call  the  clinical  laboratory  to  inquire  about  laboratory  results? 


If  answer  to  question  is  "No,"  go  to  question  «16. 


Wh.en  calling  the  clinical  laboratory  at  the  Wright-Patterson  Medical  Center,  I 


rarely 

occasionally 
about  half  the  time 
very  frequently 
almost  every  time 


get  a busy  signal. 
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16.  Once  the  telephone  in  the  clinical  laboratory  is  answered,  there  is 


1 

( 

t 

i 

I 

I 


a very  short  , 

a short  / 

a moderate  delay  m obtaining  information  that  I request, 

a long  i 

a very  long  ' 


il7.  Once  the  telephone  in  the  clinical 
I I request  is 


laboratory 


Is  answered. 


the  delay  in  obtaining  the  information  that 


much  shorter  than 
shorter  than 
about  the  same  as 
longer  than 
much  longer  than 


I would  expect  for  clinical  laboratories  of  this  kind 
(Air  Force  and  other) . 


I 


,20. 


At  the  Wright-Patterson  Medical  Center,  the  response  time  for  STAT  results  from  the  clinical 
laboratory  is 


much  shorter  than 
somewhat  shorter  than 
about  the  same  as 
slightly  longer  than 
far  longer  than 


I would  expect  for  clinical  laboratories  of  this  kind 
(Air  Force  and  other) . 


I 


Reports  from  the  clinical  laboratory  at 


the  Wriqht-Patterson  Medical  Center  have 


much  more 
somewhat  more 
all  of  the 
somewha".  less 
much  less 


clinical  information  than  (that)  I would  expect  in  lab- 
oratory reports  at  this  kind  of  hospital  (Air  Force 
and  other) . 


The  clinical  laboratory  at  this  hospital  provides 


much  mc'  * 
more 

all  of  the 
less 

much  less 


f 

statistical  information  than  (that) 


require . 


i21. 

At  the  Wright-Patterson  Medical  Center, 

clinical  laboratory 

reports  are 

much  more 

j 

I 

somewhat  more 

/ 

■ 

about  as 

readable  than 

(as)  reports  from  other  clinical 

1 

somewhat  less 

\ laboratories. 

t 

much  less 

1 

\22.  The  clinical  laboratory  at  the  Wright-Patterson  Medical  Center  makes  a 

/ 


very  large 
large 
moderate 
small 

very  small 


contribution  to  the  quality  of  patient  care. 


’23.  Overall,  I am 


mucf  more 
more 

about  as 

less 

much  less 


satisfied  with  the  clinical  laboratory  at  the  Wriqht- 
Patterson  Medical  Center  than  (as)  I have  been  with  the 
clinical  laboratory  at  other  hospitals. 


24.  At  tha  Wright-Patterson  Medical  Center,  the  reports  from  the  clinical  laboratory  contain 

more  than  all  of  j 

Call  of  ( 

most,  but  not  all,  of  > the  clinical  information  that  I would  like  from  the 

C'somewhat  lees  than  all  of  i laboratory, 

much  less  than  all  of  ' 
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1 


♦ 


I 

1 


i 

[ 


j 

( 

i 

I 

I 


r 

■t 


25.  My  relations  with  the  clinical  laboratory  personnel  at  this  hospital  are 

completely  satisfactory, 
very  satisfactory, 
reasonably  satisfactory, 
slightly  unsatisfactory, 
completely  unsatisfactory. 


The  Clinical  laboratory  at  this  hospital  provides 

/ 


much  more 
somewhat  more 
about  as  much 
somewhat  less 
much  less 


statistical  information  than  fas)  other  clinical 
laboratories . 


At  this  hospital,  laboratory  reports  are  lost 

rarely . 
occasionally, 
moderately  often, 
often . 
very  often. 

At  the  Wright-Patterson  Medical  Center,  it  is  necessary  for  me  to  repeat  requests  for  laboratory  work 

rarely . 
occasional ly . 
moderately  often, 
often, 
very  often. 

The  accuracy  of  reports  from  the  clinical  laboratory  at  this  hospital  is 

very  high, 
high, 
moderate . 
low. 

very  low. 

Laboratory  reports  at  this  hospital  are 

completely  legible, 
very  legible, 
reasonably  legible, 
slightly  illegible, 
completely  illegible. 

At  the  Wright-Patterson  Medical  Center,  the  clinical  laboratory  contributes 


much  more 
more 

about  as  much 
less 

much  less 


^ to  improved  patient  health  than  (as)  clinical  labora- 
tories at  similar  hospitals. 


The  clinical  laboratory  at  the  Wright-Patterson  Medical  Center  contributes 

/ to  the  quality  of  patient  care  than  (as)  clinical 
laboratories  at  similar  hospitals. 


much  more 
more 

about  as  much 
less 

much  less 


The  accuracy  of  reports  from  the  clinical  laboratory  at  this  hospital  is 

/ : 


much  greater  than 
somewhat  greater  than 
□ about  the  sa.'^e  as 
o somewhat  less  than 
o much  less  than 


would  expect  for  clinical  laboratories  of  this  kind 
(Air  Force  and  other). 
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34.  The  clinical  laboratory  at  this  hospital  provides  reports  that  have 

° virtually  no  errors, 

o very  few  errors. 

~ a significant  number  of  errors, 
o many  errors. 

very  many  errors. 

35.  At  the  Wriqht-Patterson  Medical  Center,  filing  of  clinical  laboratory  reports  in  patients'  charts  is 

much  more  . 

somewhat  more  / complete  than  (as)  I would  expect  at  a similar 

about  as  hospital, 

somewhat  less  i 

much  less  ' 

3fe.  My  relations  with  the  personnel  in  the  clinical  laboratory  at  this  hospital  are 

^ much  more  * 

more  ( 

as  satisfactory  than  (as)  I had  expected, 

less  V 

much  less 

37.  At  the  Wright-Patterson  Medical  Center,  the  contribution  of  the  clinical  laboratory  to  improved 
patient  health  is 

very  large, 
large . 
moderate . 
small . 
very  small. 


After  completing  thi>-  questionnaire,  please  complete  the  "Physician  Questionnaire  %2 — AFCLAS," 
which  follows. 


PHYSICIAN'S 

quest:  ONN  A IFF  ■2  — AFCLAS 

A Clinical  Laboratory  Automation  System  (AFCLAS)  has  been  installed  at  the  Wriqht-Patterson  Medical 

Center.  We  would  like  to  know  your  opinion — as 

a practicing  physician — of  the  impact  AFCLAS  has  had  on 

your  professional  activities. 

1 . AFC1.AS  has 

11 

significantly  increased 

11 

slightly  increased 

1 

(1 

had  no  effect  on 

\ the  quality  of  patient  care. 

11 

slightly  decreased 

(1 

significantly  decreased 

i 

11 

No  opinion 

1 

2.  AFCLAS  has 

n 

virtually  eliminated 

n 

slightly  reduced 

1 

11 

had  no  effect  on 

f errors  in  cl.nical  laboratory  reports. 

11 

slightly  increased 

11 

significantly  increased 

n 

No  opinion 

1 

3.  AFCLAS  has  provided 

(1 

much  more 

11 

slightly  more 

i 

n 

about  the  same  amount  of 

f statistical  information. 

n 

slightly  less 

11 

significantly  less 

11 

No  opinion 

1 

4.  AFCLAS  has 

11 

significantly  increased 

11 

slightly  increased 

1 

11 

had  no  effect  on 

\ the  amount  of  information  available  from  clinical 

11 

slightly  decreased 

/ laboratory  reports. 

I) 

significantly  decreased 

11 

No  opinion 

T 

5.  AFCLAS  is 

11 

far  superior  to 

11 

slightly  superior  to 

1 

11 

about  the  same  quality  as 

T the  previous  system. 

I) 

slightly  inferior  to 

/ 

11 

far  inferior  to 

11 

No  opinion 

I 

6.  AFCLAS  is 

11 

far  superior  to 

II 

slightly  superior  to 

i 

(1 

about  the  sane  as 

f tha  previous  system  in  providing  accurste  requasts 

11 

slightly  inferior  to 

/ for  laboratory  testa. 

II 

significantly  inferior  to 

11 

No  opinion 

1 

ANtCA 
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7. 

\FCLAS  .^4*  b«4n 

n 

far  more  effective  thar, 

[1 

slightly  more  effective 

than 

1 

ti 

about  as  effective  as 

f 

the  previous  system  in  retrieving  test 

n 

slightly  less  effective 

than 

information  on  patients. 

(I 

significantly  less  effective 

than  \ 

II 

No  opinion 

6. 

AFCLAS  responds 

to  STAT  requests  in  a way  that 

11 

significantly  improves 

i 

II 

slightly  improves 

/ 

11 

does  not  differ  from  the  previous  system 

in  providing  patient  care. 

II 

slightly  impairs 

i 

11 

significantly  impairs 

II 

No  opinion 

9. 

AFCLAS  has 

11 

significantly  improved 

II 

slightly  improved 

i 

11 

had  no  effect  on 

f 

the  ability  of  the  physicia.n  to  use  clinical 

11 

slightly  impaired 

laboratory  data. 

I! 

significantly  impaired 

II 

No  opinion 

10. 

AFCLAS  has 

11 

sign! f icantly  improved 

II 

slightly  inp>roved 

1 

the  physician's  ability  to  locate  patient's 

11 

had  no  effect  on 

f 

clinical  laboratory  data  in  the  laboratory 

11 

slightly  impaired 

report . 

11 

significantly  impaired 

II 

No  opinion 

u. 

AFCLAS  has 

11 

significantly  increased 

II 

slightly  increased 

/ 

II 

had  no  affect  on 

f 

the  accuracy  of  clinical  laboratory  reports. 

II 

slightly  decreased 

11 

significantly  decreased 

II 

No  opinion 

12. 

With  AFCLAS, 

the  clinical  laboratory 

11 

functions  much  more  smoothly 

II 

functions  slightly  more 

smoothly. 

II 

functions  about  the  same. 

11 

functions  slightly  less 

smoothly. 

II 

functions  significantly 

less 

smoothly . 

11 

No  opinion 

13. 

With  AFCLAS, 

I 

kSI 

II 

significantly  more  Anovledgeable 

II 

slightly  more  knowledgeable 

1 

II 

about  as  knowledgeable  as  in 

the  past 

/ about  my  patients. 

II 

slightly  less  knowledgeable 

1 

II 

significantly  less  knowledgeable 

II 

No  opinion 
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14. 

With  \FCLAS 

, I 

am 

n 

much  more  confident  with 

11 

slightly  more  confident  with 

i 

11 

about  as  confident  as  I was  in 

the  past  with  > the  clinical  leU>oratory  information 

11 

slightly  less  confident  with 

, that  I use  with  my  patients. 

11 

significantly  less  confident  with 

11 

No  opinion 

15. 

With  ATCLAS 

, the  format  of  information  in  lab 

reports  is 

ti 

significantly  more  i 

(1 

slightly  more  f 

I! 

no  more 

convenient 

to  use  than  was  the  previous  format. 

11 

slightly  less  1 

11 

significantly  less  ' 

11 

No  opinion 

16. 

AFCLAS  has 

11 

significantly  decreased  . 

(1 

slightly  decreased  1 

11 

had  no  effect  on  ’ 

the  time  that  I have  to  wait  for  routine  laboratory 

11 

slightly  increased  i 

reports . 

11 

significantly  increased  1 

M 

No  opinion 

17. 

AFCLAS  provides 

statistical  information  that  i 

s 

11 

significantly  more  useful  in 

i 

n 

slightly  more  useful  in 

f 

11 

no  more  useful  than  in  the  past  in 

patient  care. 

11 

slightly  less  useful  in 

11 

significantly  less  useful  in 

1 

11 

No  ofinion 

j 

18. 

AFCLAS  has 

made 

chart  reading 

I 

11 

significantly  easier. 

11 

slightly  easier. 

11 

about  as  easy  as  before  AFCLAS 

11 

slightly  mere  difficult. 

11 

significantly  more  difficult. 

11 

No  opinion 

19. 

AFCLAS  has 

11 

significantly  improved 

11 

slightly  improved  1 

11 

had  no  effect  on  ' 

telephone 

retrieval  of  test  information  by  calling  the 

11 

slightly  impaired  , 

laboratory 

11 

significantly  impaired  1 

11 

No  opinion 

20. 

APCLAS  has 

11 

significantly  increased  j 

11 

slightly  increased  f 

11 

had  no  effect  on 

the  speed 

in  retrieving  test  information  on  patients. 

11 

slightly  decreased  i 

11 

significantly  decreased  i 

11 

No  opinion 
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21. 

Arc LAS 

.has 

n 

Significantly  decreased 

II 

slightly  decreased  i 

11 

had  no  effect  on 

the  response  time  for  STAT  reports. 

M 

slightly  increased 

(1 

significantly  increased 

11 

No  opinion 

**■ 

AFCLAS 

has 

[| 

significantly  decreased 

11 

slightly  decreased 

( 

n 

M 

II 

had  no  effect  on 
slightly  increased 
significantly  increased 

the  number  of  lost  test  reports. 

11 

No  opinion 

23. 

j 

AFCLAS 

has 

11 

11 

11 

significantly  decreased 
slightly  decreased 
had  no  effect  on 

the  number  of  repeat  requests  for  laboratory  work. 

n 

slightly  increased 

11 

significantly  increased 

II 

No  opinion 

24  . 

AFCLAS 

has 

I) 

significantly  increased 

i 

I) 

slightly  increased  , 

II 

had  no  effect  on  ' 

the  legibility  of  laboratory  reports  filed  in  the 

I! 

11 

slightly  decreased 
significantly  decreased  ' 

oatient's  medical  record. 

11 

No  opinion 

25. 

AFCLAS 

has 

11 

significantly  increased 

11 

slightly  increased  I 

11 

had  no  effect  on 

the  legibility  of  laboratory  reports. 

11 

slightly  decreased  l 

11 

significantly  decreased 

11 

No  opinion 

26. 

AFCLAS 

has 

11 

significantly  decreased 

II 

•lightly  decreased 

II 

had  no  effect  on  * 

the  number  of  times  that  I call  the  laboratory  to 

11 

■lightly  increased  i 

obtain  test  results. 

II 

significantly  increased 

II 

No  opinion 

27. 

AFCLAS 

has 

11 

aignificantly  decreased 

II 

■lightly  decreaaed 

11 

had  no  effect  on 

the  total  time  that  I spend  obtaining  and  using 

11 

■lightly  increaaed 

laboratory  rsports. 

II 

significantly  increaaed 

11 

No  opinion 
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2B.  WCLAS  h«» 


tl  significantly  improved  * 

()  slightly  improved  / 

n resulted  in  no  change  in  relations  between  physicians  and  laboratory  personnel. 

(I  slightly  Impaired  i 

[I  significantly  impaired  ' 

( I No  opinion 


29.  XFC1>S  has 


(1  significantly  decreased 
[]  slightly  decreased 
()  had  no  effect  on 
[]  slightly  increased 
IJ  significantly  increased 

( ] No  opinion 


the  number  of  times  that  I visit  the  laboratory  to 
obtain  test  results. 


30.  Cumulative  laboratory  reports  provided  by  AFCLAS  are 


n much  more  useful  than  . 

(1  more  useful  than  f 

(I  about  as  useful  as  the  previous  laboratory  reports. 

()  less  useful  than  I 

11  much  less  useful  than  ’ 

()  No  opinion 


31.  AFCLAS  has 

n significantly  increased  * 
n slightly  Increased  # 

(1  had  no  effect  on  ’ the  percentage  of  laboratory  reports  filed  in  the 

(1  slightly  decreased  , patient's  medical  record. 

[]  significantly  decreased  I 

[ ] No  opiMon 


32.  My  primary  responsibility  is 

n inpatient  care. 

[]  outpatient  care. 

(1  other.  Please  specify  

33.  Were  you  assigned  to  the  Wrlght-Patterson  Medical  Center  during  March  1975? 

II  Yes 
I J No 


Please  place  your  consisted  questionnaires  into  the  accompanying  envelope,  seal  the  envelope,  and 
print  the  following  information  on  the  front  of  the  envelope: 

Name 

Department . 
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REGISTERED  NURSING  STAFF  QUESTIONNAIRE 


1 


f 


f 

I 


Instructions 

A Cllnicsl  Lsboratory  Automation  System  (AFCLAS)  has  been  installed  at  Wnght-Fatterson  Medical 
Center.  We  would  like  to  have  your  opinion — as  an  R.N. — of  the  imj-act  AFCLAS  has  had  on  your  pro- 
fessional activities. 


Please  indicate  your  response  to  each  question  by  placing  a mark  in  the  appropriate  box 

Wh.en  you  have  completed  the  questionnaire,  insert  it  in  the  accompanying  envelope,  seal  the 
envelope,  and  print  the  following  information  on  the  front  of  the  envelope: 

Name 

Department  or  Ward. 

This  information  will  be  used  only  to  ensure  that  responses  from  all  R.N.'s  at  Wright-Patterson 
Medical  Center  will  be  included  in  the  study.  The  sealed  envelope  will  be  opened  only  by  the 
research  tear  of  Analytic  Services  Inc.  (ANSFR) , and  the  information  you  provide  will  be  available 
only  to  the  team. 


1.  There  has  been 

[1  a significantly  smaller 
(1  a somewhat  smaller 
( I about  the  same- 
()  a somew.hat  greater 
(]  a significantly  greater 


number  of  telephone  calls  for  STAT  results  with  AFCLAS. 


( ] No  opinion 

2.  With  AFCLAS,  the  time  that  I spend  on  clerical  work  related  to  laboratory  reports  has  been 

(1  significantly  reduced. 
tl  slightly  reduced. 

[1  aibout  the  same  as  before  AFCLAS. 

(]  slightly  Increased. 

()  significantly  increased. 

I]  No  opinion 

3.  AFCLAS  has  made  it 


[]  much  easier 
n somewhat  easier 
[)  at  easy  as  in  the  past 
[]  somewhat  more  difficult 
[)  much  more  difficult 

{ ) No  opinion 


to  obtain  information  on  status  of  laboratory  tests 
being  performed. 


AFCLAS  has 


n significantly  decreased 
n slightly  decreased 
tl  had  no  affect  on 
[]  slightly  increased 
[]  slgnlf Icently  increased 

[]  No  opinion 


the  total  time  I spen f processing  test  requests  and 
laboratory  reports. 


AMm 
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5. 

STATS  are 

being  returned 

1 

n 

n 

II 

II 

i: 

much  faster  J 

somewhat  faster  f 

about  as  fast  as  ir.  the  past  with  AFCLAS. 

somewhat  slower  i 

much  slower  ' 

I 

II 

No  opinion 

6. 

AFCLA'*  has 

permitted  me  to  interact 

II 

much  more  effectively 

II 

slightly  more  effectively  f 

11 

about  as  effectively  as  in  the  past  with  clinical  laboratory  personnel. 

II 

slightly  less  effectively  k 

ti 

much  less  effectively 

II 

No  opinion 

7. 

AFCLAS  IS 

II 

far  superior  to  J 

II 

II 

superior  to  / i 

about  the  same  as  the  previous  system. 

! 

II 

inferior  to  1 | 

II 

far  inferior  to  ' 

11 

No  opinion 

a . 

The  collection  of  specimens  has  been 

11 

much  better  organized  » 1 

II 

better  organized  1 

II 

about  as  well  organized  as  in  the  past  with  AFCLAS.  1 

II 

less  organized  i 1 

II 

much  less  organized  i , 

II 

No  opinion  1 

I 

9. 

With  AFCLAS,  It  has  been 

II 

much  easier  i 

11 

easier  f 

II 

about  as  easy  as  in  the  past  \ to  tell  what  laboratory  woxk  is  to  be  done 

II 

more  difficult  i on  a giver,  day. 

II 

much  more  difficult  1 

II 

No  opinion 

10. 

AFCLAS  has 

mad  e 1 1 ^ 

II 

much  easier  . | 

II 

somewhat  easier  f 

II 

about  as  easy  as  in  the  past  ' to  obtain  information  ab?ut  patients' 

II 

somewhat  more  difficult  i clinical  laboratory  results. 

II 

much  more  difficult  1 

II 

No  opinion 

11. 

AFCLAS  had 

made  the  nurses'  tasks  In  ordering  laboratory  testa 

II 

ouch  easier. 

II 

somewhet  eeeler. 

(I 

about  aa  eaey  as  in  the  past . 

II 

somewhat  nore  difficult. 

II 

much  more  difficult. 

II 

No  opinion 

t 

s • 
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12.  Clinical  laboratory  reports  provided  by  WCI-AS  ar' 


3 

( 


1 


H 

] 

f 


tl  much  easier 

[)  somewhat  easier 

[]  about  as  easy  as  in  the  (vast 

(1  somewhat  more  difficult 

(1  much  more  difficult 

()  No  opin.on 

13.  With  XrCLAS,  It  IS 

()  Significantly  easier 
f/  slightly  easier 
n about  as  easy  as  ir.  the  (<ist 
I)  slightly  more  difflc.lt 
{]  significantly  eiore  difficult 

n No  opinion 


/ 

to  read. 

\ 


to  schedule  j.atients  for  labcrat'^.iy 
proced  ires . 


l4.  ATCLAS  has  made  posting  of  lir.lLjl  lai>ora*.:rv  reports  : r.  a :<it.i#-r.t's  chart 


[1  much  easier. 

()  somewhat  easier. 

I)  about  the  sam*-  as  before  AFCLAS. 
{)  somewhat  more  diffl;u'.t. 
n much  more  diffitilt 

f 1 No  op  1 r.  i on 


15.  AFCLAS  has 

9 qr.ificsrtly  i.ccreasei  j 

[]  slightly  increased  f 

(]  had  no  effect  on  tr.e  quality  of  patier.t  rare. 

(1  slightly  decreased  1 

n significantly  decreased  \ 

[)  So  opinion 


Ifc.  With  AFCI.AS.  the  number  of  t**  lephnr.*-  alls  that  I maVc  t tn*'  labcrat.i.  ; 


(]  much  smaller. 

{)  slightly  smaller, 
t]  about  the  Sdm*>  as  before  AFCUU'. . 
(1  slightly  greater, 
n much  greater. 

{]  No  opinion 


17.  My  primary 


responsibility  is 


[1  inpatient  care, 
n outpatient  care, 
n other.  Please  specify 


10.  Were  you  assigned  to  the  Wrtght-Patterson  Medi  al  Cer.tei  iuri^g  March  19'’S? 

n Yes 
[1  No 


1 


Please  place  your  completed  questionnaire  Into  the  accompanying  envelope,  seal  the  envelope,  and 
print  the  following  information  on  the  front  of  the  envelof<e; 

Name 

Department  or  Ward. 


a) 
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LAB^-IMTORY  STAFF  QUESTI^NNAIRKS 


Inst  rue  t ions 

1.  Complete  the  first  attached  questionnaire  entitled  "Laboratory  Staff  tuestionnaire  •!"  before  completing 
the  second  attached  questionnaire  entitled  “Laboratory  Staff  Questionnaire  ■2.” 

2.  Indicate  your  resjionse  to  each  question  by  placing  a mark  in  the  appropriate  block. 

3.  Insert  the  completed  questionnaires  into  the  accompanying  envelope,  seal  the  envelope,  and  print  the 
foil. wing  information  on  the  front  of  the  envelope: 

Nam<* 

Section. 

This  information  will  be  used  only  to  ensure  that  responses  from  all  laboratory  staff  at  Wright-Patterson 
.vedical  Center  will  be  included  in  the  stuay.  The  sealed  envelope  will  be  opened  only  by  the  research 
tear  of  Analytic  Services  Inc.  (ANSFR) , and  the  information  you  provide  will  be  available  only  to  the  team. 


ANSER 

April  H-P  197ft 
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f 


'i 

L 


lAH  HAT<;RY  STAKF  .■.’ESTI  'NNAJM 


r ^ " — 

St  f'  ngly 
Aiie»- 

Agr**<* 

Uncertain 

Disagree 

Strongly 

Disagree 

t" 

j l.  Durintj  ••  ir.  the  1 Ahur^f '^•ry , I d(’  tr.sny  diffejent  td<skR. 

1 

There  is  an  opt>ortunity  in  the  lab<;tatory  foi  independent  thouqht 
and  action. 

3.  In  ronttast  with  w.rxer*  in  industry,  latK.ratory  peofvle  know  the 
^ final  result*  of  their  work. 

1 

Su{ervisors  in  the  laboratory  provide  valid  ' onmer.ts  on  wy 
^•erf  -rr.ar.'  e. 

.. 

1 

The  most  importart  part  of  my  job  is  helj  im  other  f>eople 

latK-ralory  le  jet  toqether  after  wnrk  more  frequently  than 

dc.  people  in  ether  part*  ' f the  hospital. 

^ . 

My  laboratory  assiqrjnent  i-ermits  me  to  dr  many  different  test 
1 tocedures . 

A. 

My  supervisor  leaves  me  fretty  much  alone  to  jexforr  my  work 
in  the  laboratory 

1 ’■ 

I know  how  well  1 am  jerformir.q  my  job  each  day. 

^ 

IC. 

Dealing  with  other  people  in  the  laliCtatory  is  an  im[>ortant 
part  of  my  job. 

a. 

The  idborat.iry  is  a great  pla<-e  l'  develop  lose  ,*rjer;ds. 

12. 

My  laty^ratory  assignment  }>ermits  me  to  use  many  different  kinds 
of  egui|>ment  . 

13. 

Supervisors  in  the  laberatory  provide  frequent  ■itrrier.ts  on  my 
(.erf  ■ rman-  e . 

14 

r<ealing  with  peo|  le  outside  the  lalx  ratory  is  an  imf.'-rtat.t  jart 
''’I  my  "ut. 



n 

1 

1%. 

I (^rform  »asks  in  the  lab^jratory  tf.a»  are  important  but  riutside 
of  my  technical  Bi’e'.ialty. 

1 

lb. 

During  a day  in  the  laboratory.  I use  many  different  pieces  of 
equli'mer.t . 

1’. 

! am  re*p>on8ible  for  doing  lOmplete  test  procedures  in  the 
labK^ratory. 

ifl 

Whille  on  duty,  there  is  almost  always  an  op.<p>or  turn ' y to  talk 
with  laboratory  people  about  nonlaboratory  subjects. 

1 

1 

1 »’■ 

kfhen  X am  on  duty,  the  order  in  which  I perform  my  tasks, 
eacept  STATt.  is  left  up  to  me. 

i'"' 

Autooiated  eguipaaent  in  the  labfjratory  does  not  interfere  with 
*>>  decisions  alxjut  the  tasks  I perform. 

21. 

When  I cowplete  a laboratory  test  procedure,  I consider  it  a 
result  of  siy  own  efforts. 

■'.2. 

When  J start  a tsst  proesdure,  I kniiw  that  2 will  hav*  the 
opportunity  to  see  it  through  to  the  end  and  know  the  result . 

23. 

Supervisors  often  get  together  with  laboratory  people  after 
duty  houre. 

j 

1 


I 

i 
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LABORATOPY  STAFF  QUESTIONNAIPF 


1.  Th«  normal  work  week  in  this  laboratory  is 

less  than  35  hours. 

35-40  hours. 

40-45  hours. 

45-50  hours, 
over  50  hours. 

2.  During  the  fast  week,  I worked  in  the  laboratory 

less  than  35  hours. 

35-40  hours. 

40-45  hours. 

45-50  hours, 
over  50  hours. 

3.  During  the  fast  week,  I spent 


2 hours  or  less 
2-5  hours 
5-8  hours 
8-11  hours 
over  12  hours 


on  military  duties  outside  the  laboratory. 


I have  no  military  duties  outside  the  laboratory. 


4.  During  the  fast  week,  I was  on  (ass  or  leave 


5.  During  the  jast  week,  I worked 

, less  than  1 

1-3 
3-6 
6-10 
over  10 


more  hours  than  the  normal  work  week  in  the  laboratory. 


6.  In  f>erforming  my  tasks  in  the  laboratory,  I feel  that  I have 


much  more  than 
more  than 
all  of 

somewhat  less  than 
much  less  than 


the  time  that  I need  to  accomplish  my  tasks. 


17.  In  performing  my  tasks  in  the  laboratory,  I feel  that  I have 


about  the  same  amount  of 

less 

o much  less 


time  than  (that)  I would  expect  for  a laboratory  of 
this  type. 


8.  When  performing  my  tasks  in  the  laboratory,  z am  rushed 

□ rarely. 

o occasionslly . 

□ moderately  often, 
o often. 

o very  often. 


ANSBR 

April  W-P  1976 
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9.  When  performinq  my  tasks  in  the  laboratory,  I am  rushed 

much  leas  often  than 
o less  often  than  j 

about  as  often  as  ! I would  expect  in  a clinical  laboratory  of  this  kind 

more  often  than 

much  more  often  than  I 

No  opinion 

10.  The  quality  of  reports  in  this  laboratory  is 

far  superior  to 

sliqhtly  superior  to  i 

about  the  same  quality  as  f reports  for  similar  laboratories, 

j slightly  inferior  to 

far  inferior  to  i 

I Lj  No  opinion 

11.  The  time  available  for  quality  control  in  this  laboratory  is 

more  than  is  needed, 
all  that  IS  needed, 
almost  all  that  is  needed, 
sliqhtly  less  than  is  needed, 
much  less  than  is  needed. 


Quality  control  is  not  part  of  my  job. 

12.  In  laboratories  such  as  this,  there  is  typically 

” more  than  enough  time 

all  the  time  that  is  needed 
almost  as  much  time  as  is  needed 
less  time  than  is  needed 
3 much  less  time  than  is  needed 

No  opinion 

13.  Please  give  your  rank  

14.  Please  give  AFSC  number  

15.  Are  you  a student? 

Yes 
a No 

16.  Were  you  assigned  to  the  Department  of  Pathology  at  the  Wriqht-Patterson  Medical  Center  during 
March  1975? 

(I  Ve# 
n No 


Please  place  your  cotnpleted  questionnaires  into  the  accompanying  envelope,  seal  the  envelope,  and 
print  the  following  information  on  the  front  of  the  envelope: 

Name 

Section. 
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analytic:  soi*vic:c\s  iiu. 


i 
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VIRGINIA  P204I 


703  eao  3830 


3 March  1976 


LABORATORY  PATIENT  QUESTIONNAIRE 


Analytic  Services  Inc.  (ANSER) , a research  organization 
located  in  Falls  Church,  Virginia,  is  conducting  a scientific 
study  of  Air  Force  laboratories.  An  important  part  of  the 
study  is  to  find  how  people  like  you  who  use  the  laboratory 
feel  about  the  laboratory  and  its  people. 

} Your  conscientious  completion  of  the  attached  question- 

naire will  be  a major  contribution  to  the  scientific  study 
of  the  laboratory. 

Your  individual  response  to  the  questionnaire  will  be 
known  only  to  ANSER 's  staff. 

When  you  complete  the  form,  seal  it  in  the  enclosed 
envelope  and  return  it  to  the  person  who  gave  it  to  you  or 
to  the  receptionist. 

Thank  you  for  your  valuable  help  in  this  important 
study . 


Richard  Brooks,  Ph.D. 
Project  Director 
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Stronjly 

Disagree 

16  Are  you  currently  an  inpatient  at  the  Wright  Patterson  Yes 

Medical  Center’  No 

Disagree 

No 

Opinion 

Agree 

— 

Strongly 

Agree 





1 

1 

1 High  caliber  people  mork  in  the  clinical  laboratory  at  the 

\Nright  Patterson  Medical  Center 

2.  1 never  have  to  wait  a long  time  m line  at  the  clinical 

laboratory  at  the  Wright  Patterson  Medical  Center. 

3.  1 would  prefer  to  use  the  clinical  laboratory  at  the 

Wright  Patterson  Medical  Center  rather  than  any  other 
clinical  laboratory. 

4 The  clinical  laboratory  people  seem  happy  to  work  here. 

5.  1 can  completely  depend  upon  the  clinical  laboratory 

people  each  time  1 come  here. 

6.  The  atmosphere  of  the  clinical  laboratory  is  warm 
and  friendly 

7.  1 like  to  see  the  same  clinical  laboratory  people  each 

time  1 come  here. 

8.  It  seems  that  the  clinical  laboratory  people  help  each 
other  when  the  laboratory  is  busy 

9.  The  clinical  laboratory  people  do  everything  possible 
to  provide  the  best  possible  service. 

10.  1 am  never  annoyed  by  the  way  1 am  treated  by  the 

cirnical  laboratory  people  at  the  Wright  Patterson 
Medical  Center 

11.  The  cirnical  laboratory  people  treat  all  who  come  to 
the  laboratory  as  equals. 

12.  The  clinical  laboratory  people  treat  me  as  an  individual, 
not  )ust  as  another  patient  number. 

13  The  doctor  always  gets  my  test  results  that  are  needed 

and  gets  them  when  they  are  needed. 

14  The  people  in  the  clinical  laboratory  always  tell  me 

what  they  are  going  to  do  before  they  do  it. 

15.  There  is  no  better  clinical  laboratory  than  the  clinical 
laboratory  at  the  Wright  Patterson  Medical  Center. 
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APPENDIX  D 

PRETEST  FOR  EVALUATION  PLAN  FOR  THE  AIR  FORCE 
CLINICAL  LABORATORY  AUTOMATION  SYSTEM 
(AFCLAS) 
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NOTE  ON  APPENDIX  D 


In  December  1974  the  AFCLAS  Evaluation  Plan  was  pre- 
tested at  the  USAF  Medical  Center,  Keesler  AFB,  Mississippi. 
The  rationale,  general  methodology,  and  detailed  plan  for 
pretesting  the  AFCLAS  Evaluation  Plan  at  the  Medical  Center, 
Keesler  AFB,  is  contained  in  the  document  entitled  Pretest 
For  Evaluation  Plan  for  the  Air  Force  Clinical  Laboratory 
Automation  System  reproduced  in  this  appendix. 

The  pretest  considered  the  entire  process  of  data 
collection  with  considerable  emphasis  placed  on  pretesting 
the  survey  questionnaires.  The  Medical  Center  at  Keesler 
was  selected  as  the  pretest  site  because  it  is  similar  to 
the  two  AFCLAS  test  sites.  The  ANSER  study  team  and  the  four 
MET  personnel  who  collected  the  data  during  Period  X and 
Period  Y spent  7 days  conducting  the  pretest.  As  a result 
of  the  pretest,  a number  of  changes  were  made  in  the  evalu- 
ation plan  and  are  summarized  as  follows: 

• A questionnaire  for  hospital  staff  other  than 
physicians,  registered  nurses,  and  laboratory 
staff  was  dropped.  This  was  done  because  inter- 
views with  people  taking  this  questionnaire 
showed  that  they  did  not  have  enough  background 
information  to  answer  the  questions. 

• Ambiguous  questions  in  the  physician,  registered 
nurse,  and  laboratory  questionnaires  were  dropped. 

Our  interviews  with  physicians,  nurses,  and  labor- 
atory staff  showed  that  several  questions  on  each 
questionnaire  were  ambiguous,  and  since  we  did  not 
have  time  to  pretest  revised  questions,  the  only 
valid  action  was  to  drop  the  problem  questions. 
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Dropping  ambiguous  questions  did  not  seriously 
impact  the  evaluation  because  the  original 
questionnaires  contained  overlapping  questions. 

• Some  methods  for  collecting  data  inside  the 
laboratory  were  revised.  As  an  example,  "Methods 
Time  Measurement,"  an  industrial  engineering 
technique  utilizing  standard  times,  was  dropped 
in  favor  of  time  studies  preferred  by  the  MET 
team. 

• Data  collection  forms  were  revised.  The  revision 
of  data  collection  forms  was  primarily  in  layout 
to  improve  usability. 
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INTRODUCTION 


I . 


It  would  be  difficult  to  find  a general  work  on  research 
methods  that  did  not  warn  the  researcher  of  the  hazards  of 
collecting  data  from  a study  population  without  first  pre- 
testing the  study  .esign  (Reference  1).  The  warning  is  clear 
in  the  literature,  but  a detailed  description  of  pretest 
design  and  technique  is  not  to  be  found.  Writers  on  research 
methods  who  do  go  beyond  the  warning  seldom  devote  more  than 
a few  pages  to  details  (Reference  2,  3,  4).  Although  material 
on  pretesting  is  not  concentrated  in  a single  place,  basic 

concepts  can  be  gleaned  from  literature  on  research  methods. 

/ 

The  terms  "pretest"  and  "pilot  study"  often  used 

interchangeably.  John  Mudge  contends  tha't,  although  the  terms 
are  synonymous,  pretest  is  more  widely' used  in  the  United 
States  and  pilot  study  more  frequently  in  England  (Reference  5). 
Babbie,  however,  makes  a clear  distinction: 

...two  types  of  testing;,  pretests  and 
pilot  studies,  Pretesti's  . . . ref  er  to  the 
testing  of  one  or  more  aspects  of  study 
design:  the  questionnaire,  the  sample 

design,  a computer  program  for  analysis 
and  so  forth.  Pilot  studies ,,, refer  to 
miniaturized  walkthroughs  of  the  entire 
study  design.  ...  (Reference 

Babbie  further  contends  in  his  discussion  of  pretesting  and 
pilot  studies  that: 

,,. controlled  sampling  should  be  com- 
pletely ignored  at  this  point,., the 
pretest  phase  of  investigation .but , 
pilot  studies  are  another  matter.... 

(Reference  7) 

The  same  point  in  respect  to  probability  sampling  in  the 
pretest  is  made  by  Galtung,  who  states: 

...it  is  unnecessary  to  have  a statis- 
tically sophisticated  sample... in  pretest. 

(Reference  8) 
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The  pretest  design  proposed  below  will  be  as  complete 
as  the  pilot  study  proposed  by  Babbie  but  will  not  require 
a probabilistic  sample. 

Neither  of  the  sites  for  pretest  will  be  selected 
randomly,  i.e.,  probabilistically.  The  samples  will  be 
accidental  in  one  case  and  purposive  in  the  other.  The  basis 
for  pretesting  at  Malcolm  Grow  USAF  Medical  Center  at  Andrews 
AFB  was  an  accidental  sample;  the  basis  for  selection  of 
testing  at  USAF  Medical  Center,  Keesler  AFB,  is  a purposive 
sample. * 


...no  reputable  investigator  will  permit 
a questionnaire  to  go  into  the  field 
without  having  "pretested”  it  on  a range 
of  individuals  or  individuals  represent- 
ing the  principal  social  groups,  and 
experimenting  on  word  selection  with 
them. . . . (Reference  11) 

Arnold  Rose  considered  only  the  questionnaire  in  the  above 
quota  .on.  The  pretest  design  described  in  this  document 
goes  beyond  the  questionnaire  to  the  entire  data  collection 
and  analysis  process.  The  importance  of  extending  a pretest 
beyond  the  questionnaire  is  suggested  by  Galtung: 

...it  should  be  noted  that  the  pretest  is 
never  in  any  abstract  sense  only  a test 
of  the  instrument,  but  a test  of  the 
entire  process  of  data  collection  and 
even  of  the  first  steps  in  analysis.... 

(Reference  12) 

The  pretest  plan  proposed  below  conforms  to  Galtung 's  conten- 
tion that  the  entire  process  of  data  collection  should  be 
pretested. 

*...In  accidental  samples,  one  simply  reaches  out  and  talces 
the  cases  that  fall  to  hand. ... (Reference  9) 

...The  basic  assumption  behind  purposive  sampling  is  that 
~;th  good  judgment  and  appropriate  strategy  one  can  hand- 
. ry.  tne  cases  to  be  included  in  the  sample  and  thus 
• -elor>  samples  that  are  satisfactory  in  relation  to 
tf-eds ....  (Reference  10) 
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Pretesting  is  important,  but  it  is  not  the  only 
important  step  in  an  overall  study.  The  first  major  step, 
study  design,  has  been  completed  by  ANSER.  It  confoimr  to 
Simon's  admonition  to  researchers  to: 

...think  at  length,  think  in  detail,  think 
about  everything  before  collecting  data. 

This  must  begin  to  sound  like  a broken 
record  but  it  is  worth  the  repetition. 

Time  and  money  spent  collecting  data  be- 
fore you  have  done  all  the  thinking  will 
be  expensive  in  the  short  and  long  run. 

But  of  course  the  future  cannot  be  seen 
perfectly. ... (Reference  13) 

No  scientist,  no  matter  how  much  he  thinks,  can  perfectly 
predict  what  will  happen  when  the  design  is  implemented  in 
the  field.  The  basic  purpose  of  pretest  is  to  provide  a more 
valid  prediction  of  what  will  happen  when  data  are  collected 
in  the  field  and  what  will  happen  when  data  collected  are 
analyzed.  As  Babbie  points  out, 

...Inevitably  the  researcher  fails  to 
recognize  in  advance  all  the  decisions  he 
must  make ....  (Reference  14) 

The  research  staff  and,  if  possible,  the  interviewers 
should  participate  in  all  phases  of  the  pretest  (Reference  15) 
The  staff  will  obtain  a feel  for  the  data  that  can  be  obtained 
only  by  participation  in  the  data  collection  process. 

Much  of  the  literature  on  pretesting  is  concerned  with 
the  purpose  of  pretesting  questionnaire  and  interview 
schedules.  Selltiz,  et  al.,  comment  on  the  pretest  of 
questionnaires : 

...The  pretest  is  a tryout  of  the  question- 
naire to  see  how  it  works  and  whether 
changes  are  necessary  before  the  start  of 
full  scale  study.  The  pretest  provides  for 
catching  and  solving  unsolved  problems  in 
the  administration  of  the  questionnaire, 
such  as  phrasing  and  sequencing  of 
questions  or  its  length. ... (Reference  16) 
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They  further  point  out  that: 


...questions  that  seem  clear  and  straight- 
forward to  the  research  staff  may,  on  a 
trial  testing,  prove  difficult  to  compre- 
hend, or  ambiguous,  or  simply  not  productive 
of  useful  inf ormation . . . . (Re terence  17) 

In  a different  context,  though  directly  applicable  to 
the  pretest,  S.  L.  Payne  speaks  of  the  critical  issue  in 
questionnaire  design: 

• , o to  make  sure  the  particular  issue  which 
the  questioner  has  in  mind  is  the  particu- 
lar issue  on  which  the  respondent  gives 
an  answer ....  (Reference  18) 

In  designing  a study,  the  researcher  usually  has  clearly 
in  mind  the  information  he  wants  to  obtain,  but  when  he 
develops  the  schedules  they  reflect  his  own  perception  of  the 
meaning  of  words.  Words  mean  one  thing  to  the  researcher  and 
another  to  the  respondent.  Researchers  are  bound  by  their 
own  experience.  In  studying  a professional  environment  other 
than  his  own,  the  researcher  tends  to  impute  his  meaning  of 
words  to  the  respondent.  As  Oppenheim  states: 

...Pilot  work  is  needed  to  ensure  that 
we  use  terms  that  are  like  those  that 
the  respondents  use  themselves  and  have 
roughly  similar  meaning  to  most  of  them — 
and  to  us .... (Reference  19) 

An  important  part  of  the  pretest,  thus,  is  to  determine  what 
words  and  terms  mean  to  the  population  studied  in  the  survey. 

The  interview  questionnaire  creates  the  same  kinds  of 
problems  as  the  written  questionnaire.  It  is  possible  in  an 
interview  to  probe,  of  course,  but  there  are  still  problems. 
Kahn  and  Cannell  contend  that: 

. . .No  matter  how  astute  the  interviewer 
has  been  in  wording  questions,  in  develop- 
ing the  proper  sequence  of  questions  and 
in  the  design  of  the  questionnaire,  he 
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needs,  wherever  possible,  to  try  it  out 
before  collecting  the  actual  interview 
....(Reference  20) 

Oppenheim  also  writes  of  the  significance  of  pretesting 
to  identify  ambiguities: 

Pilot  work  can  be  the  greatest  help  in 
devising  the  actual  wording  of  questions 
and  it  operates  as  a healthy  check, 
since  ambiguities  may  lurk  in  the  most 
unexpected  quarters ....  (Reference  21) 

The  emphasis  on  question  wording  is  apparent  in  the  above. 
Mudge  goes  beyond  this  to  contend  that  a pilot  study  is 
primarily  an  exercise  in  answering  questions  for  the 
researchers  (Reference  22). 

Most  writers  would  agree  with  Selltiz  et  al.  that  the 
objective  of  pretesting  is  to  test  all  of  the  data  collection 
techniques  to  be  used  in  the  survey: 

...Much  difficulty  can  be  avoided  by 

carefully  pretesting  the  techniques  to 

be  used,  to  ensure  that  they  will 

collect  the  information  needed. ... (Reference  23) 

The  sections  of  the  pretest  design  that  follow  provide 
a detailed  description  of  what  is  to  be  done  in  pretesting 
questionnaires  and  schedules. 

Some  general  remarks  on  the  method  of  pretesting  are  in 
order  at  this  pointo  The  two  sites  for  the  pretest  are  to 
be  selected  on  the  basis  of  1)  an  accidental  sample  and  2) 
a purposive  sample.  Malcolm  Grow  Hospital  was  selected  for 
pretest  of  activity  charting  and  some  other  tasks,  such  as 
the  identification  of  the  reports  generated  in  a clinical 
laboratory.  It  was  chosen  primarily  for  the  convenience  of 
the  MET  team  stationed  at  Andrews  AFB  and  the  proximity  of 
the  base  to  others  concerned  with  the  study  (an  accidental 
sample) . Since  Malcolm  Grow  Hospital  will  be  the  site  of 
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the  second  AFCLAS  installation,  it  would  be  inappropriate 
to  pretest  some  of  the  questionnaires  and  schedules  at  that 
base.  The  Medical  Center,  Keesler  AFB,  was  selected  as  the 
pretest  site  on  the  basis  of  the  similarity  of  the  popula- 
tion of  the  pretest  site  to  the  population  of  the  survey 
sites  (Reference  24).  The  Medical  Center,  Keesler  AFB,  is 
similar  to  the  Malcolm  Grow  Hospital  and  USAF  Medical  Center, 
Wright-Patterson  AFB  in: 

• Number  of  procedures  per  year  in  the  clinical 
laboratory 

• Characteristics  of  the  medical  staff  (teaching 
hospital) 

• Size  of  the  hospital 

• Population  served. 

The  sample  was  thus  a purposive  sample. 

The  data  collection  section  in  the  pretest  design  relates 
data  collection  techniques  to  each  hypothesis  to  be  tested. 

In  the  pretest,  each  technique  will  be  tested  and  evaluated. 
Section  III,  below,  describes  in  detail  the  method  for  the 
pretest  of  data  collection  techniques  to  be  used  in  the 
study.  In  addition  to  techniques,  the  forms  used  in  data 
collection  will  be  pretested  and  evaluated  (Reference  25). 

The  study  design  requires  samples  of  a population,  e.g., 
physicians,  nurses,  patients;  samples  of  objects,  e.g., 
patient  records,  laboratory  requests;  and  samples  of  activi- 
ties, e.g.,  time  to  retrieve  information,  number  of  calls  to 
the  laboratory.  The  samples  of  populations  will  follow 
established  methods  and  will  not  require  pretesting.  The 
sampling  of  patients'  records  will  require  pretesting  to 
determine  the  sample  size  for  the  survey  and  to  estimate  the 
time  required  to  collect  data.  The  sample  of  activities  does 
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not  present  any  methodological  problem,  but  it  is  difficult 
to  anticipate  the  specific  problems  that  might  exist  in 
recording  data. 

The  pretest,  as  noted  above,  does  not  require  a proba- 
bilistic sample  (References  26,27).  However,  as  a part  of  the 
pretest  of  schedules,  physicians  will  be  interviewed  at  the 
pretest  site  to  obtain  their  observations  on  the  schedules. 

In  order  to  reduce  the  time  required  for  interview,  each 
physician  will  be  asked  to  comment  on  four  randomly  selected 
questions  from  the  schedule.  Howard  Schuman  refers  to  this 
as  the  random  probe  technique  (Reference  28).  The  random 
probe  technique  will  be  discussed  in  more  detail  in  Section 
II  below. 
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II 


PRETEST  OF  SCHEDULES 


The  rationale  for  pretesting  of  schedules  was  discussed 
in  Section  I,  above.  In  this  section  the  details  of  data 
collection  for  pretesting  schedules  will  be  described. 

A.  Selection  of  Respondents 

Schedules  have  been  developed  for  generating  information 

from: 

• Physicians 

• Nurses 

• Staff  (outpatient  and  nursing  unit) 

• Laboratory  staff 

• Medical  records  staff 

• Patients. 

A list  of  physicians  assigned  to  the  hospital  will  be 
obtained.  The  list  should  include  the  following  information: 

• Name 

• Primary  assignment  within  the  hospital 

• Career  or  noncareer. 

A list  of  all  nurses  assigned  to  the  hospital  will  be 
obtained.  The  list  should  include: 

• Name 

• Primary  assignment  within  the  hospital 

• Whether  civilian  or  military. 

A list  will  be  obtained  of  all  personnel,  except 
physicians  and  nurses,  assigned  to  the  hospital  outpatient 
department  and  inpatient  nursing  units.  The  list  should 
include  the  following: 
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Name 


• 

• Status  (officer,  airman,  civilian) 

• Specialty. 

A list  of  all  personnel  assigned  to  the  laboratory  will 
be  obtained.  The  list  should  include  the  following: 

• Name 

• Status  (officer,  airman,  civilian) 

• Specialty. 

A list  of  names  of  all  medical  records  personnel  will 
be  obtained. 

Patient  respondents  will  be  identified  by  a sample  of 
every  nth  (n  to  be  determined)  patient  who  enters  the 
laboratory  during  selected  hour(s)  and  day(s).  The  days 
and  hours  for  sampling  will  be  determined  by  stratifying  the 
hours  into  before  noon  and  after  noon.  If  it  is  assumed 
that  there  are  differences  in  population  among  the  days  of 
the  week,  the  week  should  be  stratified  accordingly. 

The  study  design  assumes  that  use  and  perception  of  the 
laboratory  by  physicians  vary  with  (and  only  with)  primary 
assignment,  specifically,  inpatient  and  outpatient,  and 
whether  the  physician  is  career  or  noncareer.  The  pretest 
physician  population  will  be  stratified  into: 

• Career,  outpatient  assignment 

• Career,  inpatient  assignment 

• Noncareer,  outpatient  assignment 

• Noncareer,  inpatient  assignment. 

A sample  from  each  stratum  will  be  selected  for  inter- 
view only  if  significant  differences  among  strata  are 
observed  in  the  responses.  The  interview  will  be  for  purposes 
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of  testing  the  validity  of  the  questions  on  the  schedule. 

It  will  be  necessary,  therefore,  to  identify  the  physician 
with  the  completed  questionnaire.  There  is  a risk  that 
there  will  be  bias  because  of  identification.  To  avoid  or 
reduce  such  bias,  the  following  procedure  will  be  followed. 
Each  questionnaire  will  be  numbered,  but  no  other  identifi- 
cation will  be  on  the  questionnaire.  The  respondent  will 
place  his  completed  questionnaire  in  an  envelope  and  print 
his  name  on  the  outside.  ANSER  personnel  will  compile  a 
list  of  questionnaire  identification  numbers  and  the 
associated  name  of  the  respondent.  This  list  will  be  kept 
confidential,  known  only  to  the  staff  and  not  disclosed  to 
others . 

The  nurses  will  be  stratified  into  inpatient  and  out- 
patient nurses.  An  approximately  equal  number  will  be 
selected  from  each  stratum.  The  number  selected  from  each 
stratum  should  be  at  least  50.  The  nurses  to  be  interviewed 
will  be  selected  in  the  same  manner  as  physicians.  As  was 
noted  in  Section  I,  the  sample  need  not  be  random. 

The  outpatient  staff  and  inpatient  nursing  unit  staff 
respondents  should  consist  of  the  entire  population  if  50 
or  less  or  a sample  of  approximately  50  if  the  population  is 
significantly  greater  than  50,  i.e.,  60  or  more. 

The  clinical  laboratory  staff  should  be  stratified  into 
officer,  airman,  and  civilian.  Since  the  reliability  and 
validity  of  the  schedule  administered  to  the  clinical  labora- 
tory staff  has  been  tested  previously,  only  two  or  three 
respondents  need  be  selected  from  each  stratum. 

The  medical  records  staff  population  will  not  be 
administered  by  a schedule  but  will  be  interviewed.  Since 
the  number  in  this  population  is  expected  to  be  small,  it  is 
necessary  only  to  identify  elements  in  the  population. 
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In  summary,  the  following  steps  will  be  taken  to  select 
respondents : 

• Obtain  lists  of  the  following: 

Physicians,  including:  name,  primary  assignment 

in  the  hospital,  whether  career  or  noncareer 

- Nurses,  including:  name,  assignment  in  hospital, 

and  status  (civilian  or  military) 

Outpatient  staff  and  inpatient  nursing  unit  staff 
(excluding  physicians,  nurses),  including  name 
and  status  (civilian  or  military) 

/ - Laboratory  staff,  including:  name,  status 

(officer,  airman,  civilian) , and  specialty 

Medical  records  staff,  including  name  only 

• Sample  time  periods  for  administering  schedules  to 
outpatients 

• Stratify  physicians  by  career,  noncareer,  outpatient, 
inpatient  assignment 

• Identify  elements  to  be  surveyed 

• Identify  elements  to  be  interviewed. 

B.  Sampling  of  Questions  or  Schedule  That  Will  Be  Subject  of 
Physician  and  Nurse  Interview 

The  random-probe  technique  (Reference  1)  will  be  used 
to  identify  the  questions  to  be  asked  each  physician  and 
nurse  element  in  the  study.  The  use  of  the  random-probe 
technique  will  reduce  the  time  required  to  interview  each 
element . 

Each  element  in  the  sample  will  be  interviewed  for 
response  to  four,  and  only  four,  questions.  The  method  for 
selecting  the  questions  is  described  in  the  Technical  Note 
appended  to  this  section. 
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C.  Types  of  Interview 

Interviews  will  be  conducted  with  a sample  of  each 
category  of  personnel  (except  the  medical  records  personnel) 
to  determine  the  validity  of  the  questionnaires. 

Each  of  these  interviews  will  be  composed  of  three 
basic  types  of  interviews: 

• Focused  standardized  open-ended 

• Standardized  open-ended 

• Nondirective. 

The  focused  interview  was  developed  by  Merton,  Fiske, 
and  Kendall  (Reference  2) . In  a focused  interview,  the  inter 
viewer  knows  the  experience  of  the  respondent,  and  the 
interviewer  focuses  attention  on  the  aspects  of  experience 
significant  for  goals  of  the  project.  In  this  study  the 
interviewer  will  know  in  advance  the  response  that  the  respon 
dent  has  previously  given  to  the  questions  that  will  be  the 
subject  of  the  interview. 

Specifically,  interviews  with  physicians  and  nurses 
will  focus  on  the  following  subjects: 

• The  reason  for  response  to  a given  question,  e.g., 
the  reason  physician  responded  "no  opinion"  to 
question  number  5 

• How  the  respondent  perceived  the  meaning  of  the 
question,  e.g.,  when  asked  about  the  accuracy  of 
laboratory  reports,  did  the  respondent  answer  in 
terms  of  technical  accuracy  of  the  laboratory  or 
the  accuracy  of  the  report  with  respect  to  the 
respondent's  related  activity? 

• Perception  of  terminology,  e.g„,  was  the  terminology 
too  technical,  not  technical  enough,  not  professional 
terminology? 
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• Was  the  meaning  of  words  precise  or  ambiguous  to 
the  respondent? 

• Was  the  response  subjective  or  objective,  i.e., 

was  response  based  on  the  feelings  of  the  respondent 
_ or  upon  specific  knowledge? 

• The  importance  of  the  subject  matter  of  the  question 
to  the  respondent,  e.g.,  did  the  respondents  consider 
ease  of  calling  the  laboratory  significant  in  their 
instrumental  activity? 

Standardized  questions  will  be  used  in  exploring  each 
of  these  subjects.  In  a standardized  interview  the  same 
questions  are  asked  of  all  respondents.  The  wording  and 
order  in  which  the  questions  are  asked  are  the  same  for  all 
respondents.  The  response  to  the  initial  question  in  each 
case  will  be  open-ended.  The  respondent  replies  exactly  as 
he  thinks. 

The  open-ended  response  will  be  followed  by  nondirective 
probing.  Nondirective  interviewing  was  developed  from  the 
work  of  Carl  Rogers  in  client-centered  counseling  (Reference  3) . 
In  client-centered  therapy,  the  client  determines  the  subjects 
to  be  discussed.  The  therapist  makes  only  neutral  responses 
to  the  client's  remarks,  e.g.,  "Oh,"  "Uhm,"  "I  see."  The 
technique  has  been  modified  and  adapted  to  the  information 
gathering  interview.  Instead  of  the  client  (respondent) 
determining  what  will  be  discussed,  the  interviewer  focuses 
on  matters  to  be  discussed.  This  is  called  controlled  non- 
directive interviewing.  The  neutral  role  of  the  interview*^! 
remains  in  controlled  nondirective  interviewing.  The 
viewer  introduces  the  problem  or  question  and  then  r*”  • 
neutral  until  he  has  gathered  the  information  ,ies;:i- 
the  problem  or  question,  at  which  time  he  in*-r 
problem  or  question. 
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A variation  of  controlled  nondirective  interviewing 
referred  to  in  the  literature  as  controlled  nondirective 
probing  will  be  used  in  interviewing  nurses,  physicians, 
laboratory  personnel,  and  outpatient  staff. 

In  interviewing  the  medical  records  staff,  only  con- 
trolled nondirective  interviewing  will  be  used.  In  inter- 
viewing patients,  nondirective  probing  will  be  used. 

Thus,  the  techniques  to  be  used  in  interviewing  each 
population  are; 


Population 

Physicians 

Nurses 

Outpatient  staff 

Laboratory  staff 

Medical  records  staff 
Patients 


Interview  Technique 

Focused  standardized  open-ended; 
standardized  open-ended;  non- 
directive probing 

Focused  standardized  open-ended; 
standardized  open-ended;  non- 
directive probing 

Focused  standardized  open-ended; 
standardized  open-ended;  non- 
directive probing 

Focused  standardized  open-ended; 
standardized  open-ended;  non- 
directive probing 

Controlled  nondirective 

Nondirective  probing 


D.  Categories  of  Questions  and  Questionnaires 

There  are  many  categories  of  questions  and  questionnaires 
identified  in  the  literature.  The  categories  described  below 
are  only  those  that  apply  to  the  AFCLAS  study  and  pretest. 
Questions  and  questionnaires  are  categorized  in  the  three 
ways  that  are  applicable  to  this  study;  method  of  adminis- 
tration, content  of  questions,  types  of  questionnaire. 
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The  administration  of  questionnaires  will  take  two  forms: 
self-administered  and  gro^^-administered. 

Self-administered  questionnaires  typically  are  given  to 
the  respondent  by  someone  not  a part  of  the  research  team. 

The  respondent  completes  the  questionnaire  and  returns  it  to 
a designated  person  or  place.  In  the  present  study,  the 
patient  questionnaire  will  be  passed  out  to  every  nth  (n  to 
be  determined)  patient  who  enters  the  clinical  laboratory 
during  periods  defined  by  the  pretest  staff.  If  there  is  an 
operations  or  reception  desk  in  the  laboratory,  a person 
regularly  assigned  to  that  function  will  pass  out  the  question- 
naire. It  will  be  collected  before  the  patient  leaves  by 
another  person  assigned  to  the  laboratory. 

The  technique  that  will  be  used  in  the  pretest  interviews 
of  physicians,  nurses,  outpatient  staff,  and  laboratory  staff 
is  a group-administered  questionnaire.  The  questionnaires  will 
be  administered  to  each  group  of  personnel  in  a general  meeting. 

The  content  categories  of  questions  applicable  to  the 
pretest  are: 

• Facts  - A question  about  a fact  is  directed  to  a 
person  who  should  know  the  fact,  e.g.,  a member 

of  the  laboratory  staff  might  be  asked  how  many  hours 
overtime  he  worked  last  week. 

• Belief  about  what  the  facts  are  - The  respondent  is 
asked  questions  not  to  find  out  objective  data  but 
what  the  respondent  believes  the  facts  are. 

Selltiz  et  al.  state  the  typical  example  of  this: 

...one  does  not  measure  the  temperature  of 
a room  by  asking  the  people  in  it  how  hot 
they  believe  it  is.  But  if  one  is  inter- 
ested in  the  subjective  experience  of  tem- 
perature under  varying  conditions,  ne  may 
follow  precisely  this  procedure  of  asking 
people  how  hot  they  believe  the  room  is.... 

(Reference  4) 
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There  are  a number  of  factual  questions  included  in 
the  AFCLAS  schedules  that  ask  questions  that  are 
factual,  but  the  interest  is  in  perception  of  these 
facts,  e.g.,  perception  of  accuracy  of  laboratory 
reports. 

• Feelings  - It  is  often  important  not  only  to  know 

how  a person  perceives  of  facts,  but  also  his  feelings 
toward  these  facts.  In  the  physicians'  schedules 
there  are  questions  that  relate  to  how  the  physician 
feels  about  the  clinical  laboratory. 

• Standards  of  action  - These  are  "should"  or  "ought  to" 
questions.  The  researcher  is  concerned  with  what 

the  respondent  thinks  should  or  ought  to  be  done 
about  something.  For  example,  one  of  the  questions 
that  has  been  proposed  to  ask  physicians  in  the 
survey  is,  "What  should  be  done  to  improve  the 
clinical  laboratory?" 

The  AFCLAS  study  will  use  a single  type  of  questionnaire 
that  will  be  of  the  standardized  type  and  will  provide  the 
respondent  with  fixed  alternatives. 

E.  Probing 

No  matter  how  much  care  and  skill  the  researcher  exerts 
in  the  development  of  a questionnaire,  there  is  always  a 
possibility  that  his  questions  will  be  misunderstood  or  mis- 
interpreted by  respondents  in  the  field. 

The  purpose  of  pretest  is  to  perfect  the  questions  in 
the  field  for  the  survey  to  the  extent  possible.  The  probe 
is  a technique  by  which  prepared  questions  are  supplemented 
during  the  interview  process. 


205 


The  probe  can  be  in  any  form  of  interview  or  in  combi- 
nations of  forms.  The  probe  in  the  pretest  will  be  a combi- 
nation of  standardized,  focused,  and  nondirective. 

The  interview  for  pretest  of  the  questionnaires  will 
focus  on  the  six  response  variables  listed  in  Section  C, 
above,  and  the  content  categories  listed  in  Section  D,  above. 
The  interviewer  will  be  provided  with  standardized  open  ended 
questions,  which  he  will  ask  the  respondent.  When  the  respon- 
dent replies,  the  interviewer  will  use  a controlled  non- 
directive probing  technique. 

The  interviewer  will  be  provided  with  the  questions  he 
will  discuss  with  the  physician,  the  category  of  each  question 
based  on  the  intent  of  the  question  as  designed,  and  the 
standardized  question  he  will  use  to  begin  the  probe  of  each 
variable. 

The  following  is  a brief  illustration  of  the  technique. 

After  introducing  himself  and  setting  the  scene 

the  interviewer  continues  — 

I.  (Interviewer) — Dr.  Able,  in  response  to  question  5 

you  checked  "No  opinion."  Would  you  tell  me 
something  about  your  reasons  for  this?* 

R.  (Respondent) — Oh,  I don't  know.  I think  it's  hard 

to  speak  of  accuracy  in  the  laboratory» 

lo  — It's  hard  to  speak  of  accuracy? 

R.  — ^Yes,  it's  hard  when  you  can't  be  sure  what 
standards  there  are.  Are  they  my  standards  or 
laboratory  standards? 

I.  — I see.  Could  you  tell  me  more? 


*The  questions  with  the  asterisk  are  standardized  and  will 
be  provided  to  the  interviewer.  The  interviewer  dialogue 
other  than  the  standardized  question  is  nondirective. 
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Ro  — Well,  I have  my  standards  and  the  laboratory 
might  have  theirs. 

I • Um J 

R.  — I don't  know  which  standards,  so  I checked  no 
opinion. 

I.  — I see. 

I.  — How  did  you  perceive  the  meaning  of  the  question?* 

Nondirective  probing  takes  the  form  of  neutral  statements. 
The  neutral  statement  may  be  a repetition  of  what  the  respon- 
dent said;  e.g.,  in  the  above  interview,  R.  says,  "I  think 
it's  hard  to  speak  of  accuracy  in  the  laboratory"  and  I.  says, 
"It's  hard  to  speak  of  accuracy?" 

The  neutral  statement  may  be  in  the  form: 

— I see 

— Oh 

— Tell  me  more 
— Uh  huh 
' Um . 

F.  Interviewer  Selection 

There  are  no  definitive  rules  for  the  selection  of  inter- 
viewers. However,  there  are  certain  general  principles  that 
can  be  derived  from  the  experience  of  survey  researchers  and 
from  sociological  theory.  The  principles  relate  to  the 
dynaunics  of  the  interview. 


♦After  this  standardized  question  the  interviewer  would 
probe  again  until  he  was  satisfied  that  he  had  an  answer 
to  the  question.  He  would  continue  the  interview  in  this 
way  until  the  six  response  variables  had  been  discussed 
with  the  respondent. 


207 


The  interview  is  an  encounter  between  two  strangers  who 
thus  become  a two-member  (dyadic)  group.  The  interaction  in  a 
group  thus  formed  is  somewhat  different  from  groups  formed 
for  purposes  other  than  interviewing,  but  there  are  character- 
istics common  to  all  groups. 

When  two  strangers  meet,  the  decision  to  interact  is 
based  on  their  perception  of  each  other.  Initial  influences 
are  observable  characteristics  of  each  other,  e.g.,  dress, 
grooming,  language,  sex,  ethnicity.  From  variables  such  as 
these  we  may  deduce  other  characteristics,  such  as  social 
status  and  education.  If  interaction  begins,  each  member 
of  the  group  may  estimate  social  variables  from  the  inter- 
action process.  In  the  interview  encounter  these  variables 
will  influence  not  only  whether  a group  will  form,  i.e., 
whether  the  respondent  will  agree  to  be  interviewed,  but  also 
the  validity  and  reliability  of  the  information  obtained 
from  the  interview. 

As  Sjoberg  and  Nett  point  out,  the  reliability  and 
validity  of  data  generated  by  interviews  depends  upon  rapport, 
which  is  directly  related  to  roles  and  statuses  of  the  inter- 
viewer and  the  respondent: 

...The  interview,  we  must  remember,  involves 
a complex  set  of  interactions  between  per- 
sons who  occupy  particular  statuses  in  the 
social  order.  Consequently,  the  reliability 
and  validity  of  the  data  obtained  depend 
not  merely  upon  the  merits  of  the  question- 
naire but  also  upon  the  degree  of  rapport, 
or  trust,  that  develops  in  the  interaction 
situation.  Thus  if  we  are  to  understand  the 
structured  interview,  we  must  analyze  the 
status  and  roles  of  the  principal  parties 
concerned. ... (Reference  5) 

Kahn  and  Cannell  relate  the  status  variables  of  inter- 
viewer and  respondent  to  intervening  psychological  variables 
and,  in  turn,  to  behavior  factors.  The  model  they  propose. 
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reprinted  below,  emphasizes  the  psychological  nature  of  the 
interview.  The  interaction  in  the  model  is  not  explained  by 
action  but  by  psychological  factors.  (Note  that  the  arrows 
do  not  connect  interviewer  and  respondent.)  A sociologist 
would  present  a quite  similar  model  but  omit  the  intervening 
variables.  We  would  postulate  their  existence  but  would  not 
analyze  the  psychological  intervening  variables. 


INTERVIEWER 


RESPONDENT 


5 


T 


Background  characteristics: 

Background  characteristics: 

Age 

Age 

Education 

Education 

Socioeconomic  status 

Socioeconomic  status 

Race 

Race 

Religion 

\ / 

Religion 

Sex 

\ / 

Sex 

Etc . 

\/ 

Etc. 

Z 


Psychological  factors: 

/ \ 

Psychological  factors: 

Perceptions 

/ \ 

Perceptions 

Attitudes 

Attitudes 

Expectations 

Expectations 

Motives 

Motives 

Behavioral  factors: 

Errors  in  asking  questions 
Errors  in  probing 
Errors  in  motivating 
Errors  in  recording 

responses 


)ehaviorai  factors: 
Responses  to  questions 
(Adequate-inadequate) 
(Accurate- inaccurate) 


Source:  Reproduced  from  Kahn  and  Cannell  (Reference  6) 


The  model  appears  to  emphasize  the  possible  errors  made 
by  the  interviewer.  The  errors  can  be  made  by  the  respondent 
as  well  as  the  interviewer.  As  Sjoberg  states: 
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...It  is  clear  that  the  status  of  the 
investigator  is  a significant  variable 
affecting  the  quality  of  data  that  a 
respondent  will  provide.  More  generally, 
we  advance  the  proposition  that  the 
potential  impact  of  the  interviewer  on 
the  interviewee,  judged  by  the  reliabil- 
ity and  validity  of  the  data  collected, 
is  greater  in  highly  stratified  social 
systems  than  in  the  more  loosely 
organized  or  relatively  homogeneous 
ones. ... (Reference  7) 

The  study  of  the  military  hospital  brings  the  researcher 
into  two  of  the  most  highly  stratified  systems  in  our  society 
the  hospital  and  the  military  service.  If  Sjoberg  is  right, 
more  status  problems  can  be  expected  than  in  surveys  of  a les 
stratified  population. 

Williams  contends  that  difference  in  status  perceived  by 
the  respondent  results  in  the  respondent  biasing  his  response 

...The  greater  the  disparities  among 
status  characteristics  between  inter- 
viewer and  respondent  the  greater  is 
the  pressure  felt  by  the  respondent 
to  bias  his  responses .... (Reference  8) 

Benny  and  Hughes,  in  their  study  of  the  influence  of 
status  upon  interaction  in  the  interview,  were  concerned 
primarily  with  a high  status  interviewer  interviewing  lower 
status  respondents.  They  contend  that  such  status  differ- 
ences can  be  minimized  by  training: 

...Whatever  actual  inequalities  of  sex, 
status,  intelligence,  expertness  or 
physique  exist  between  the  parties 
should  be  muted.  Interviewer  training 
consists  very  largely  of  making  inter- 
viewers aware  of  the  kinds  of  social 
inequalities  with  which  respondents 
are  likely  to  be  concerned  and  of 
teaching  them  how  to  minimize  them.... 

(Reference  9) 
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They  appear  to  be  less  confident  of  the  effect  of  train- 
ing when  a low  status  interviewer  interviews  a high  status 
respondent.  In  fact,  they  suggest  that  the  respondent  will 
respond  subjectively  not  to  the  interviewer  but  to  someone 
else.  They  follow  the  statement  quoted  above  with: 

...But  what  happens  when,  as  increasingly 
happens,  a run  of  the  mill,  middle  class 
interviewer  encounters  a member  of  some 
financial,  intellectual  or  political  elite? 

Our  own  impression  is  that  such  respondents 
contrive  to  reestablish  equality  in  the 
interview  by  addressing  themselves  subjec- 
tively, not  to  the  actual  interviewer,  but 
to  the  study  director  or  even  his  sponsor.... 
(Reference  10) 

Williams  states  categorically  that: 

...The  greater  the  social  distance  between 
the  interviewer  and  the  respondent  the 
greater  the  likelihood  of  bias,  all  else 
being  equal.  (Reference  11) 

Dohrenwend  and  her  associates  conclude  from  a review  of 
studies  of  the  effect  of  status  differences  between  inter- 
viewer and  respondents: 

...that  lack  of  status  homophyly  between 
interviewer  and  respondent  produces 
biased  responses .... (Reference  12) 

The  few  quotations  from  the  extensive  literature  con- 
cerning the  influence  of  status  differences  on  validity  and 
reliability  of  interview  generated  data  are  significant. 

It  appears  that  the  best  way  to  minimize  bias  is  through 
selection  of  interviewers.  Denzin  observes  that: 

. . .A  degree  of  muted  equality  permeates 
the  interview. .. yet  this  sense  of  perfect 
equality  seldom  fits  any  interview.  High 
status  respondents  may... talk  past  the 
interviewer ... .Thus  if  the  fiction  of 
equality  is  to  be  realized,  a fit  in  back- 
ground and  status  between  interviewer  and 
subject  must  be  maximized. ... (Reference  13) 
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Kahn  and  Cannell  make  an  even  stronger  statement: 

...Such  characteristics  as  age,  sex, 
education,  race,  religion  and  socio- 
economic status  can  be  influenced,  for 
the  most  part,  only  by  selection..,. 

It  is  impossible  to  bring  about  changes 
in  background  characteristics  by  train- 
ing or  by  practice. ... (Reference  14) 

Background  characteristics  of  respondents  to  be  inter- 
viewed at  the  AFCLAS  pretest  site  vary  widely,  but  the 
respondents  can  be  placed  in  categories  relative  to  back- 
ground characteristics. 

In  the  social  system  of  the  military  hospital  the  follow- 
ing status  groupings  can  be  identified: 

• Physicians 

• Nurses 

• Other  officers 

• Airmen 

• Civilians 

The  above  categories  are  not  in  a hierarchical  order, 
since  high  ranking  nonmedical  officers  may  have  higher 
status  than  low  ranking  physician  officers.  It  is  unnecessary 
for  purposes  of  the  AFCLAS  pretest  to  place  status  groupings 
in  a hierarchical  order.  It  is  sufficient  to  recognize  that 
if  bias  is  to  be  avoided,  selection  or  assignment  of  inter- 
viewers must  be  based  upon  some  "fit"  of  interviewers  and 
respondents. 

Social  status  appears  to  be  the  most  significant  variable 
in  interviewer  selection.  Education  may  be  a variable,  but 
it  is  probably  correlated  with  social  status.  There  is  no 
reason  to  believe  at  this  stage  in  the  development  of  the 
pretest  plan  that  other  background  variables  such  as  age, 
sex,  race,  or  religion  will  significantly  affect  the  validity 
and  reliability  of  interview  generated  data. 
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TECHNICAL  NOTE  FOR  SECTION  II 


ASSIGNMENT  OF  QUESTIONS  TO  INTERVIEWS  FOR  PRETEST  OF  SCHEDULES 


Let  n = number  of  questions  on  the  schedule 


p = number  of  questions  to  be  tested  in  each  interview 

2 = number  of  interviews  in  which  each  question  will 
be  tested 


Then 

I = number  of  interviews  required  = 


p|  n 


if  p-f2n 


The  number  p of  questions  per  interview  should  be  chosen  so 
that  p I 2n  and  preferably  so  that  p|n.  However,  the  primary 
consideration  in  selecting  p is  how  well  that  number  of 
questions  asked  each  respondent  will  serve  the  purposes  of 
the  test. 


Each  of  the  I interviews  will  be  randomly  assigned  p 
questions  by  the  following  procedure: 


1.  Generate  2 random  permutations  of  the  numbers  1 through  n. 

Let  these  be  q , q , . . . , q and  r , r , . . . , r . 

1 ^ n 12  n 

2.  Assign  question  q / q / q to  the  first  interview, 

1 2 P 

“^p+i'  ^p+2'  •••'  ‘^jP  second  interview  and  so  on  until 

n questions  have  been  assigned. 


3.  Case  1„  p|n. 

Repeat  2.  for  the  second  permutation  beginning  with  the 

r , . . 


assignment  of  questions  r , 

^ + 1.  When  the  process  is  completed  there  will  be  exactly 


r to  interview  number 
P 


* x|y  is  read  x divides  y,  i.e.,  there  is  an  integer  z 
such  that  x*z  = y.  x^y  is  read  x does  not  divide  y. 

t [x]  = the  greatest  integer  t x. 
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9 n 

p questions  assigned  to  each  of  I = — interviews. 

Case  2.  p|'n. 

When  n questions  have  been  assigned  there  will  be 
interviews  with  p questions  assigned  and  the 

interview  will  have  been  assigned  the  n - p 


+ 1st  *-  ■ 
questions 


/ • • • » q 


n 


Complete  the  assignment  of  p 


questions  to  interview 
P -fn  - p 


i]) 


+ I as  follows:  Assign  the  first 

questions  of  the  second  permutation  unless 

If  q . = r . 

It 


q.  = r.  for  some  q and  r in  the  assignment! 

ID  ^ -“i  3 

for  some  q and  r assigned  in  this  way,  choose  a random 
starting  point  in  the  second  permutation  and  assign  the 


next  p - I n - p 


questions  to  the 


+ 1st  interview 


unless  q^  = r^  for  some  q and  r in  the  assignment.  If 
there  is  some  q^  = r^  in  the  assignment,  choose  another 
random  starting  point  in  the  second  permutation.  Repeat 
this  process  until  an  assignment  is  made  with  no  q^  = r^. 
Continue,  assigning  the  next  p questions  in  the  permutation 
to  the  next  interview  and  so  on  as  in  2o  following  assign- 
ment of  r^  with  r^  if  r^  was  not  the  starting  point,  until 
all  n questions  have  been  assigned. 

Subcase  2o1q  p|2n. 

In  this  case  the  procedure  described  will  result  in  p 
questions  assigned  to  each  of  I = -^  interviews. 

Subcase  2.2  prf 2n. 

In  this  case,  the  procedure  described  will  result  in 


P 


interviews  with  p questions  assigned  to  them  and  one 

f2n1 


interview  with  2n  - p 


< p questions  assigned 
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Conduct  this  interview  with  only 

questions  or  randomly  select  for  the  interview 
an  additional  p - f questions,  ignoring  questions 
selected  that  are  already  included  in  the  assignment. 


Example:  n = 13 

P = 3 

3f2.13  so  I 


+ 1 = 


26 

m 


+ 1 


8 + 1 


9. 


The  two  random  permutations  are: 


A.  13  2 3 5 4 1 8 7 6 11  12  9 10 

B.  13  10  1 8 9 7 11  12  6 5 3 2 4 

Start 


Interview 

Questions 

1. 

13, 

2,  3 

2. 

5, 

4,  1 

3. 

8, 

7,  6 

4. 

11, 

12,  9 

5. 

10, 

5,  3 

6. 

2, 

^4,  13 

7. 

10, 

1,  8 

8. 

9, 

7,  11 

9. 

12, 

6,[T] 

% 


('^  indicates  end  of  permutation  A,  3-fl3  so  this  is  Case  2. 
Continuing  with  first  number  of  permutation  B would  give  the 
assignment  10,  13,  10.  A random  starting  point  selected 
from  a table  is  10.) 

(Last  question  assigned,  number  1,  was  selected  at  random. 

An  initial  random  selection  of  question  6 was  ignored,) 
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III.  PRETEST  OF  DATA  COLLECTION  METHODS 


This  plan  specifies  procedures  for  pretesting  the  methods 
of  data  collection  to  be  used  in  Period  X of  the  AFCLAS  eval- 
uation. Data  collection  methods  and  forms  to  be  tested  as 
well  as  the  test  procedures  are  given  for  each  hypothesis  ex- 
cept those  for  which  survey  schedule  responses  will  provide 
the  information.  The  pretest  of  survey  schedules  is  treated 
separately  in  Section  II  of  the  pretest  plan.  Parts  of  the 
pretest  of  data  collection  method  can,  and  in  some  cases 
should,  take  place  before  or  after  the  planned  pretest  week 
at  Keesler  AFB. 

Hypothesis  1;  Time  spent  producing  administrative  reports 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

The  data  collection  concerning  time  spent  preparing 
administrative  reports  is  to  consist  of  identifying  the 
reports  and  all  tasks  involved  in  preparing  them  and  then 
measuring  or  collecting  data  to  estimate  the  time  spent  by 
each  category  of  personnel  for  each  task.  Pretest  data  will 
be  collected  for  two  administrative  reports: 

• A report  required  by  an  authority  external  to  the 
laboratory,  such  as  the  monthly  workload  report 

• An  internal  management  report,  such  as  a quality 
control  report. 

Methods  to  be  used  in  measuring  the  time  spent  on  specific 
tasks  associated  with  this  hypothesis  depend  on  the  details 
of  the  tasks.  These  details  will  be  determined  before  or 
during  the  pretest  week.  However,  it  is  expected  that  most 
of  these  methods  will  be  similar  to  those  described  herein 
for  other  hypotheses. 


!;■  FFECEDIM5  PAGE  BLAN1L.N0T  FIIWiCp 
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Procedure 


1.  MET — identify  all  tasks  contributing  to  the  report 
(Form  Attachment  A) 

2.  ANSER — specify  method  of  measuring  or  collecting 

data  to  estimate  time  spent  by  each  personnel  cate- 

* 

gory  for  each  task. 

3.  MET — carry  out  data  collection  for  at  least  one 
task  for  each  method  specified.  Proposed  methods 
include : 

• Charting  and  application  of  standard  time  units 

• Estimation  by  personnel  involved  of  the  time 
they  spend  (Form  2,  Attachment  A) 

• Recording  by  personnel  involved  of  the  time 
they  spend  (Form  2,  Attachment  A) 

• Work  sampling  observation, 

4.  ANSER/MET — review  results,  identify  measurement 
problems,  and  revise  methods. 

5.  Continue  in  3,  4 loop  until  satisfactory  measure- 
ment methods  are  established, 

6.  MET — obtain  supervisor  estimates  of  the  fraction  of 
total  task  time  contributed  by  each  personnel  cate- 
gory for  the  tasks  identified  for  which  it  is 
appropriate  (Form  3,  Attachment  A)t 


* When  the  method  is  specified,  other  data  collection  and 
pretest  tasks  will  be  identified,  e g,,  developing  charts, 
counting  forms, 

t This  step  is  specified  in  the  pretest  procedures  for 
numerous  hypotheses.  After  it  has  been  carried  out 
successfully  for  several  hypotheses,  it  need  not  be 
carried  out  for  the  remaining  hypotheses. 


Hypotheses  2,  3,  4,  5:  Telephone  Inquiries 

2.  The  number  of  telephone  inquiries  to  the 
laboratory  central  site  (the  reception  desk 
or  central  computer  site)  will  change,  for  an 
equal  number  and  distribution  of  test  requests 
and  an  equal  number  of  telephone  lines  and 
personnel  at  the  central  site. 

3.  Time  spent  responding  to  telephone  inquiries 
received  by  the  laboratory  central  site 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests  and  an  equal  number 
of  telephone  lines  and  personnel  at  the 
central  site. 

4.  Time  spent  on  the  telephone  for  inquiries  to 
the  laboratory  by  personnel  outside  the  labora- 
tory will  change,  for  an  equal  number  and  dis- 
tribution of  test  requests  and  equal  number  of 
telephone  lines  and  personnel  at  the  central 
site . 

5.  The  distribution  by  location  and  type  of  caller 
of  telephone  inquiries  to  the  laboratory  central 
site  will  change  for  an  equal  number  and  distri- 
bution of  test  requests,  and  an  equal  number  of 
telephone  lines  and  personnel  at  the  central 
site. 

Data  for  these  hypotheses  will  be  collected  by  observation 
during  specified  time  intervals.  The  methods  must  be  tested 
during  the  pretest  week.  (Additional  trials  at  the  test  sites 
themselves  prior  to  Period  X are  desirable  and  will  probably 
be  required.  The  two  test  sites  are  organized  differently  for 
answering  the  inquiries,  centralized  at  one  site,  and  decen- 
tralized at  the  other.) 

Procedure 


Case  1 — for  centralized  system  (Andrews)  and  entry  point 
in  decentralized  system  (Wright-Patterson) : 

1.  ANSER — ^randomly  select  intervals  for  observation. 
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c MET — observe  area  (reception  desk  or  other  area 
where  inquiries  are  initially  received)  and  record 
telephone  inquiry  data  (Forms  4 and  5,  Attachment  A) 
during  the  selected  intervals.* 

The  following  information  is  to  be  recorded  for  incoming 
calls  observed  at  the  central  area: 

• The  time  that  the  call  begins  and  the  time  that  it 
ends  or  is  referred 

• Whether  or  not  the  call  is  an  inquiry  (An  inquiry 

is  any  request  for  test  results  or  information  about 
test  status.) 

• For  an  inquiry,  whether  the  caller  is  a physician, 
nurse,  other  member  of  the  hospital  staff,  a patient 
(or  relative  of  a patient) , or  other  person 

• For  an  inquiry,  the  caller's  location:  a specific 

ward  or  clinic,  or  "other" 

• For  an  inquiry,  the  status  when  the  call  ends: 
complete,  referred  (and  where  referred) , or  labora- 
tory will  call  back 

• The  personnel  category  of  the  person  who  receives 
the  call. 

The  following  information  will  be  recorded  for  response 
calls  (calls  made  from  the  central  area  back  to  the  location 
initiating  the  inquiry)  from  the  central  area  that  are 
observed : 

• The  time  that  the  call  begins  and  the  time  that 
it  ends 


* One  question  to  be  answered  in  the  pretest  is  whether  a 
single  observer  can  record  data  for  these  hypotheses  as 
well  as  for  Hypothesis  35  during  the  same  observation 
interval . 
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• Estimate  by  the  person  calling  of  the  time  that  was 
spent  retrieving  the  information  requested 

• The  personnel  category  of  the  person  responding 

• The  location  called. 

3.  MET/ANSER — meet  to  review  experience  during  obser- 
vation, identify  problems  and  revise  forms  and 
methods, 

4.  Repeat  2,  3 loop  with  discussion  once  daily  or  after 
observation  periods  as  necessary/possible. 

Case  2 — for  inquiries  answered  outside  the  central  area 

in  a decentralized  system. 

1.  Assume  all  referred  inquiries  go  to  a laboratory 
department. 

2.  ANSER — randomly  select  intervals  for  observation, 
stratified  by  laboratory  department. 

3.  MET^-observe  and  record  telephone  inquiry  data  for 
specified  department  and  time  interval  (Forms  4 and 
5,  Attachiment  A). 

The  information  to  be  recorded  for  calls  to  and  from  the 
laboratory  section  is  the  same  as  that  already  specified  for 
calls  to  and  from  the  laboratory  central  area  (see  Case  1, 
step  2)  . 

4.  MET/ANSER — jointly  review  observation  experience, 
identify  problems,  and  revise  forms  and  methods. 

5.  Repeat  3,  4 loop  once  daily  or  following  observation 
periods  as  necessary  or  possible. 

Hypothesis  6;  Time  required  for  filing  records  of  laboratory 
results  in  laboratory  files  will  change,  for 
an  equal  number  and  distribution  of  test 
requests . 
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The  method  of  measuring  time  spent  in  filing  will  be 
activity  charting  and  application  of  time  standards.  The 
data  collection  includes  identification  of  result  records 
that  are  filed  (slips,  worksheets,  etc.);  activity  charting 
and  verification  of  associated  times;  and  collecting  the 
information  required  in  using  the  charts  to  estimate  time 

spent.  Charting  will  not  necessarily  be  done  during  the 

★ 

pretest  week;  however,  methods  of  counting  items  filed  will 
be  tested  during  the  pretest  week  for  each  filing  task 
identified. 

Procedure 

1.  MET — chart  request/result  slip  filing.  If  filing 
is  decentralized,  determine  the  number  of  different 
charts  needed  and  prepare  at  least  one.  (If  differ- 
ences in  the  tasks  introduce  major  differences  in 

the  charting,  more  than  one  chart  should  be  prepared.) 
Include  associated  activity  times  and  frequencies 
with  charts,  and  indicate  how  they  were  derived. 

2.  ANSER/MET — review  charts  and  associated  times,  and 

identify  problems  and  needed  revisions. 

■'< 

3.  MET — revise  charts  and  times. 

4.  Continue  2,  3 loop  until  charts  and  associated  times 
are  finalized. 

5.  MET — record  number  prepared  daily  of  each  of  the 
following  kinds  of  items  filed: 


* The  general  chart  development  procedure  outlined  for 

Hypothesis  6 is  to  be  used  for  all  charts  required.  Charting 
done  before  or  during  the  pretest  week  will  test  this  proce- 
dure. Charting  is  included  in  the  pretest  procedures  for 
numerous  hypotheses.  However,  when  charting  for  several 
hypotheses  has  been  successfully  accomplished,  it  need  not 
be  carried  out  in  the  pretest  for  the  remaining  hypotheses. 
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• Request  slips  by  department  (obtain  from  each 
department  at  the  end  of  each  day  using  Form  6, 
Attachment  A) 

• Others  not  yet  identified  (counting  most  of 
these  items  will  be  covered  in  procedures  for 
other  hypotheses) . 

6.  MET/ANSER — review  experience  in  obtaining  information 
with  the  methods  and  forms  indicated  in  5,  identify 
problems,  and  revise  the  methods  and  forms  as 
required . 

7.  Continue  5,  6 loop  until  methods  and  forms  are 
finalized. 

8.  MET — obtain  supervisor  estimates  of  fraction  of 
taslc  time  contributed  by  each  category  of  personnel 
involved  (Form  3,  Attachment  A). 

Hypothesis  7;  Time  required  to  file  laboratory  reports  in 
outpatient  medical  records  will  change,  for 
an  equal  number  and  distribution  of  test 
requests . 

The  comments  and  procedures  for  Hypothesis  6 apply 
except: 

• Result  slip  filing  is  the  only  filing  task. 

• Items  to  be  counted  are  result  slips  by  day  and  by 
department  for  outpatients  (Form  6,  Attachment  A). 

Hypothesis  8i  Time  required  to  file  laboratory  reports  at 
the  nursing  stations  will  change,  for  an 
equal  number  and  distribution  of  test 
requests . 

The  comments  and  procedures  for  Hypothesis  6 apply 
with  the  following  additional  remarks: 

• Filing  will  include  all  activities  carried  out  in 
preparing  slips  to  be  entered  in  the  patient's  record. 


• A separate  chart  may  be  needed  for  each  nursing  unit 
or  a single  chart  may  apply  to  two  or  more  units — 
this  is  to  be  determined  in  the  pretest. 

• Items  to  be  counted  are  result  slips  by  day  and  by 
department  for  inpatients  (Form  6,  Attachment  A). 

Hypothesis  9;  Time  required  for  preparing  laboratory  clinical 
forms  will  change,  for  an  equal  number  and 
distribution  of  test  requests. 

The  method  to  be  used  in  measuring  the  time  spent  in 
these  tasks  will  be  activity  charting  and  application  of  time 
standards.  Charting  specified  in  the  pretest  procedure  will 

•k 

not  necessarily  be  done  during  the  pretest  week,  but  methods 
for  the  counting  or  sampling  that  is  required  to  apply  the 
standards  will  be  tested  then.  Data  collection  for  this 
hypothesis  includes  identification  of  all  laboratory  clinical 
forms;  activity  charting  and  verifying  time  standards;  and 
counting  or  measuring  workload  elements  to  which  the  standards 
apply.  (This  hypothesis  applies  only  to  activities  inside 
the  laboratory.) 

Procedure 

1.  MET — prepare  charts  with  associated  times  for  prep- 
aration of  the  following  laboratory  clinical  forms: 

• Request  slips  (Preparation  includes  verifying 

and  completing  request  data  and  entering  results.) 

• One  kind  of  worksheet. 

• One  log  book. 

2.  MET — obtain  copies  of  a sample  of  completed  forms  of 
each  kind  (to  be  used  to  establish  the  number  of 
characters  per  unit)  and  other  data  in  support  of 
time  standards  (Form  7,  Attachment  A). 

* See  note  on  page  224. 
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3.  MET/ANSER — review  charts  and  associated  times, 
identify  problems  and  needed  revisions. 

4.  MET — ^revise  charts  and  times. 

5.  Continue  2,  3,4  loop  until  charts  and  times  are 
finalized. 

6.  MET — count  request  slips  by  department  by  day  (Form  6, 
Attachment  A) . 

7.  MET — count  each  kind  of  worksheet: 

• Count  total  number  for  1 month  in  files  if 
retained  (Form  8,  Attachment  A). 

• Record  number  worksheets  prepared  by  day,  for 
each  kind  of  worksheet  (Forms  7,  9,  Attachment  A). 

8.  MET — measure  log  book  entries  (Measure  will  depend 
on  time  standards  to  be  used) . Measures  to  be 
tested  include: 

• Measure  inches  of  completed  logs  (time/inch 
standard  applies) . 

• Count  log  book  entries — patients  or  quality 
control  sample  runs,  etc.  (time/entry  standard 
applies) . 

• Other  kinds  identified  and  suggested. 

9e  MET — count  or  measure  other  items  (identified  in 

the  course  of  the  pretest  to  which  standards  apply) 

* 

(Form  8,  Attachment  A). 


* Some  initial  work  on  charting  at  the  test  sites  is  needed 
to  determine  what  methods  must  be  tested  for  measuring 
items  to  which  standards  apply. 


MET/ANSER — review  experience  in  obtaining  infor- 
mation with  the  methods  and  forms  indicated  in  6,  7, 
8,  and  9,  identify  problems,  and  revise  the  methods 
and  forms  as  required. 

Continue  6,  7,  8,  9,  and  10  loop  until  methods  and 
forms  are  finalized. 

MET — obtain  supervisor's  estimates  of  fraction  of 
total  task  time  contributed  by  each  category  of 
personnel — for  each  kind  of  laboratory  form  and 
each  laboratory  section  (Form  3,  Attachment  A). 


Hypothesis  10;  Time  spent  compiling  College  of  American 

Pathologists  (CAP)  standard  workload  figures 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

Pretest  for  Hypothesis  1 will  serve  as  the  pretest  of 
this  hypothesis. 

Hypothesis  11:  Time  required  for  labeling  specimens  will 

change,  for  an  equal  number  and  distribution 
of  test  requests. 

In  investigating  this  hypothesis,  it  will  be  assumed 
that  it  is  sufficient  to  consider  only  the  time  that  is  spent 
in  recording  the  necessary  information  on  the  labels.  The 
method  to  be  used  in  measuring  this  time  will  be  activity 
charting  and  application  of  time  standards.  The  charting 

phase  of  this  pretest  procedure  will  not  necessarily  be 

* 

done  during  the  pretest  week,  but  methods  of  counting  or 
estimating  the  number  of  labels  prepared  will  be  tested 
then. 


* See  note  on  page  224. 
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Procedure 


1.  MET — prepare  chart  of  specimen  labeling  and  associated 
time  spent  with  the  information  recording  tasks. 

2.  MET — determine  all  information  items  recorded  on  a 
label  and  provide  other  data  supporting  the  times 
specified. 

3.  MET/ANSER — review  charts  and  times,  identify  problems 
and  needed  revisions. 

4.  MET — revise  charts  and  times. 

5.  Continue  loop  until  charts  and  times  are  finalized. 

6.  The  number  of  labels  prepared  must  be  counted  or 
estimated.  Methods  to  be  tried  include: 

(a)  Have  the  number  of  tubes  per  venipuncture 
recorded  when  the  number  of  venipunctures  is 
recorded.  MET — collect  number  daily,  or  less 
often  as  appropriate  (Forms  8 and  9, 

Attachment  A) . 

(b)  ANSER— design  a sampling  scheme  to  estimate  the 
number  of  tubes  per  request  slip;  MET— observe 
tubes  per  slip  for  specified  sample  of  slips. 

(c)  MET — obtain  number  of  tubes  and  number  of  slips 
from  each  phlebotomist  on  return  from  collection 
rounds;  have  personnel  in  drawing  room  record 
number  of  tubes  and  slips;  MET^-collect  record 
daily. 

(d)  For  specimens  other  than  tubes  of  blood,  MET — 
obtain  labels  per  test,  per  request  or  other 
known  relationship;  provide  information  for 
selecting  an  alternative  method  if  not  feasible. 

MET/ANSER — select  most  feasible  method  and  apply  it. 
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7.  MET/ANSER-review  experience  with  forms  and  the 
methods  of  counting  or  estimating  selected,  identify 
problems  and  revise  forms  and  method  or  select 
another  method  for  trial. 

8.  Continue  6,  7 loop  until  forms  and  counting  or 
estimating  method  are  finalized. 

9.  MET-obtain  supervisor  estimates  of  fraction  of 
total  task  time  contributed  by  each  category  of 
personnel  (Form  3,  Attachment  A). 

Hypothesis  12:  Time  required  for  preparing  lists  for  specimen 

collection  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

The  method  of  measuring  the  time  spent  in  this  task  will 
be  activity  charting  and  application  of  time  standards.  As 

stated  for  previous  hypotheses,  the  test  of  chart  development 

★ 

may  be  independent  of  the  pretest  week.  However,  methods 
of  counting  or  measuring  items  to  which  standards  apply  will 
be  tested  in  the  pretest  week. 

Procedure 

1.  The  procedure  for  chart  development  is  the  same  as 
that  described  previously  for  other  hypotheses. 

2.  Items  to  which  the  time  standards  will  be  applied 

I 

I are  request  slips  for  inpatient  specimen  collection 

,1  rounds.  Methods  to  be  tested  for  counting  or  esti- 

mating the  number  of  such  slips  include: 

• MET — count  slips  for  inpatient  specimen  collection 
immediately  before  collection  rounds  (Forms  8,  9, 
Attachment  A) . 

* See  note  on  page  224. 
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• Collection  teams  record  number  of  slips  for  each 
regular  collection  trip.  MET — obtain  after  each 
collection  trip.  If  Hypothesis  11,  Procedure 
6(c)  is  selected  for  counting  tubes  labeled, 
this  method  will  also  be  selected  (Form  8, 
Attachment  A) . 

• When  inpatient  slips  are  counted  daily,  allocate 
them  by  stamped  time  of  arrival  to  the  various 
collection  trips.  This  implies  the  assumption 
that  all  inpatient  specimens  are  obtained  during 
collection  rounds,  and  this  is  not  the  case. 

(Modify  Form  6,  Attachment  A to  include  this 
information. ) 

MET/ANSER — select  most  feasible  method  and  apply  it. 

3.  MET/ANSER — review  experience  with  forms  and  the  method 
of  counting  or  estimating  the  number  of  slips  for 
inpatient  specimen  collection;  identify  problems 

and  revise  forms  and  method  or  select  another  method 
for  trial  , 

4.  Continue  2,  3 loop  until  forms  and  method  of  counting 
or  estimating  are  finalized. 

5.  MET — obtain  supervisor's  estimates  of  the  fraction 
of  total  tas)c  time  contributed  by  each  category  of 
personnel  (Form  3,  Attachment  A). 

Hypotheses  13  and  14:  Recording  Results  of  Online  and  Offline 

tests 

13.  Time  required  for  recording  results  of  tests 
that  will  be  online  with  AFCLAS  will  change, 
for  an  equal  number  and  distribution  of  test 
requests . 

14.  Time  required  for  recording  results  of  tests 
that  will  be  offline  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 
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The  time  required  for  these  tasks  is  included  in  the 
time  to  prepare  laboratory  clinical  forms,  which  is  measured 
for  Hypothesis  9.  The  following  pretest  procedure  pertains 
to  additional  information  needed  and  will  not  be  carried  out 
during  the  pretest  week. 

Procedure 


MET — identify  equipment  and  associated  tests  that  will 
be  online  and  offline  with  AFCLAS.  Also,  determine  what 
fraction  of  tests  of  a given  kind  will  be  online  or  offline. 

Hypotheses  15  through  19;  Computer  Data  Entry 

15.  Time  required  for  entering  laboratory  requests 
by  mark-sense  card  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 

16.  Time  required  for  keyboard  entry  of  patient 
name  and  identification  at  the  reception  desk 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

17.  Time  required  for  keyboard  entry  of  patient 
name  and  identification  in  the  registrar's 
office  will  change,  for  an  equal  number  and 
distribution  of  test  requests. 

18.  Time  required  for  entering  laboratory  test 
results  by  mark- sense  card  will  change,  for  an 
equal  number  and  distribution  of  test  requests 

19.  Time  required  for  keyboard  entry  of  free-text 
laboratory  test  results  will  change,  for  an 
equal  number  and  distribution  of  test  requests 

No  data  for  these  hypotheses  will  be  collected  in  Period 
X.  Pretest  of  data  collection  methods  will  be  done  prior  to 
Period  Y,  but  after  Period  X. 

Hypotheses  20  and  21;  Review  and  Certification  of  Results 

20.  Time  required  for  supervisor's  review  and 
certification  of  results  and  worksheets  will 
change,  for  an  equal  number  and  distribution 
of  test  requests. 
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21.  Time  required  for  technician's  review  and 

certification  of  results  and  worksheets  will 
change,  for  an  equal  number  and  distribution 
of  test  requests. 

The  method  of  measuring  review  and  certification  time 
will  be  activity  charting  and  application  of  time  standards. 
Charting  specified  in  the  pretest  procedure  will  not 
necessarily  be  done  during  the  pretest  week,  but  methods 

of  counting  or  estimating  the  number  of  items  certified 

* 

will  be  tested  then. 

Procedure 

1.  MET — chart  the  review  and  certification  procedure 
for  supervisors  or  technicians.  The  procedure  for 
developing  charts  is  the  same  as  that  described 
previously  for  other  hypotheses.  (Procedures  for 
certification  may  differ  in  the  various  laboratory 
sections.  Therefore,  part  cf  the  task  of  charting 
will  be  determining  whether  the  differences  are 
great  enough  to  require  separate  charts  for  each 
department  or  whether  a single  chart  will  suffice 
for  several  departments) « 

2.  MET — identify  all  items  that  are  certified  by  super- 
visors and  those  certified  by  technicians  to  which 
the  time  standards  apply. 

3.  Procedures  for  testing  methods  of  counting  request/ 
result  slips,  worksheets  and  log  book  entries  have 
been  specified  for  other  hypotheses  (See  pretest 
procedures  for  Hypotheses  6,  7,  8,  and  9;  and  Forms 
6,  8,  and  9,  Attachment  A). 


* See  note  on  page  224. 


4.  ANSER — specify  methods  and  forms  for  counting  items 
certified  that  have  not  previously  been  considered. 

5.  MET — apply  methods  specified  in  4. 

6.  MET/ANSER — review  experience  in  using  the  methods 
and  forms  specified,  identify  problems,  and  revise 
the  forms  and  methods. 

7.  Continue  5,  6 loop  until  methods  and  forms  are 
finalized. 

Hypotheses  22,  23,  and  24;  Quality  Control 

22.  Time  spent  on  mathematical  analysis  for 
quality  control  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 

23.  The  number  of  times  statistical  analyses  of 
quality  control  sample  results  are  performed 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

24.  The  kinds  of  statistical  analyses  of  quality 
control  sample  results  will  change,  for  an 
equal  number  and  distribution  of  test  requests. 

The  method  that  will  be  used  to  measure  time  spent  for 
calculations  (Hypothesis  22)  is  activity  charting  and  appli- 
cation of  time  standards.  The  frequency  and  kinds  of 
analyses  (Hypotheses  23  and  24)  will  be  obtained  as  part  of 
the  charting  procedure.  Analyses  of  quality  control  sample 
results  are  tasks  contributing  to  quality  control  reports 
that  are  included  in  Hypothesis  1.  Charting  specified  in 
the  pretest  procedure  will  not  necessarily  be  done  during  the 

it 

pretest  week. 


* See  note  on  page  224. 
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Procedure 


1.  MET — chart  quality  control  analysis  activity.  The 
procedure  for  developing  charts  is  the  same  one 
previously  described. 

2.  MET — prepare  a list  of  kinds  of  analyses  performed. 

3.  MET — determine  frequency  of  calculations.  Methods 
to  be  tested  include: 

• Obtain  a schedule  for  the  task  from  the 
laboratory  supervisor  if  one  exists;  determine 
whether  calculations  are  carried  out  each  time 
they  are  scheduled. 

• Personnel  involved  record  number  of  times  each 
kind  of  calculation  is  performed  (this  may  be 
done  if  few  people  are  involved  and  calculation 
is  infrequent)  (Form  2,  Attachment  A).  MET — 
observe  to  ensure  that  record  is  kept  and 
collect  weekly  (Form  9,  Attachment  A). 

4.  MET/ANSER — select  most  feasible  method  and  MET — 
apply  it. 

5.  MET/ANSER — review  experience  in  applying  the  method 
selected  for  determining  calculation  frequency; 
identify  problems  and  revise  method  and  forms  used 
or  select  another  method  for  trial. 

6.  Continue  4,  5 loop  until  method  and  forms  are 
finalized. 

7.  MET — obtain  supervisors'  estimates  of  the  fraction 
of  task  time  contributed  by  each  category  of 
personnel  (Form  3,  Attachment  A),. 
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Hypotheses  25  and  26:  Analysis  of  Patient  Results  by 

Population 

25.  The  number  and  kinds  of  statistical  analyses  of 
patient  results  by  population  will  change,  for 

an  equal  number  and  distribution  of  test  requests. 

26.  The  time  required  for  statistical  analyses  of 
patient  results  by  population  will  change,  for 

an  equal  number  and  distribution  of  test  requests. 

Since  this  task  is  performed  infrequently,  it  may  not 
be  possible  to  collect  hard  data  on  it.  Information  provided 
by  laboratory  officers  will  have  to  be  relied  upon.  Infor- 
mation will  be  obtained  during  the  pretest  week.  Additional 
information  from  the  test  sites  will  be  needed  to  make  a 
decision  about  the  treatment  of  the  hypotheses  in  the 
evaluation. 

Procedure 

1.  MET— obtain  from  test  sites  lists  of  analyses  performed, 
their  frequencies,  and  schedules  if  they  exist. 

2.  MET — obtain  a list  of  kinds  of  analyses  performed  at 
the  pretest  site  and  their  frequency.  Obtain  a 
schedule  if  any.  If  analyses  are  scheduled  during 
the  pretest  week,  observe  whether  they  are  done 

3.  MET — obtain  from  personnel  involved  an  estimate  of 
time  spent  for  analyses  carried  out  in  the  past,  or 
a record  of  time  actually  spent  during  the  pretest 
week  (Form  2,  Attachment  A). 

4.  MET/ANSER — review  information  obtained  and  experience 
in  obtaining  it. 

5.  ANSER — use  information  from  AFCLAS  test  sites  and  the 
pretest  site  to  decide  how  the  hypotheses  will  be 
treated  in  the  evaluation. 
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Hypothesis  27;  Time  spent  on  calculations  and/or  conversions 
from  raw  to  clinical  values  for  test  reports 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

The  method  to  be  used  in  measuring  time  spent  in  doing 
calculations  for  test  reports  will  be  activity  charting  and 
application  of  time  standards.  Charting  specified  in  the 
pretest  procedure  will  not  necessarily  be  done  during  the 
pretest  week. 

Procedure 

1.  MET — prepare  list  of  all  calculations  done  by  tech- 
nicians for  test  reports.^ 

2.  MET/ANSER — select  one  kind  of  calculation  for  charting. 
(If  charting  is  to  be  done  during  the  pretest  week, 
verify  that  the  calculation  is  done  at  the  pretest 
site. ) 

3.  The  procedure  for  developing  charts  and  standards  is 
the  same  as  described  for  previous  hypotheses. 

4o  Items  to  be  counted  for  applying  the  standard  times 
are  tests  for  which  calculations  are  done.  The 
method  of  counting  tests  has  been  specified  for 
previously  discussed  hypotheses  (see  Hypotheses  6, 

7,  8,  9,  and  Form  6,  Attachment  A). 

5.  MET — obtain  supervisor  estimates  of  the  fraction  of 
total  time  for  each  kind  of  calculation  contributed 
by  each  personnel  category  (Form  3,  Attachment  A). 


* See  note  on  page  224. 

t To  be  done  prior  to  the  pretest  week  if  possible 


Hypothesis  28;  Time  spent  on  operation  of  the  computer  by 
the  laboratory  staff  will  change.* 

Data  collection  for  this  hypothesis  will  only  be  required 
at  one  of  the  two  test  sites,  Wright-Patterson  AFB,  during 
Period  X.  The  procedure  for  collecting  this  data  cannot  be 
tested  during  the  pretest  week. 

Procedure 

1.  MET — identify  computer  operation  tasks  at  the  Wright- 
Patterson  AFB  clinical  laboratory. 

2.  Possible  methods  for  measuring  the  time  spent  include 

• Determine  number  of  full-time  equivalent 
personnel  assigned. 

• Use  activity  charting  and  application  of 
time  standards. 

• Have  personnel  involved  record  time  spent 
(Form  2,  Attachment  A). 

MET/ANSER — select  most  feasible  method  and  MET— carry 
out  specified  measurement  at  Wright-Patterson  AFB. 

3.  MET/ANSER — review  experiences  in  using  the  selected 
method;  identify  problems  and  revise  method  or 
select  another  method  for  trial. 

4.  Continue  2,  3 loop  until  a method  of  measurement  is 
chosen. 

Hypothesis  29;  Kinds  and  numbers  of  laboratory  clinical 

forms  used  will  change,  for  an  equal  number 
and  distribution  of  test  requests. 

Data  collection  for  this  hypothesis  is  included  in  that 
for  Hypotheses  1 and  9. 

* Time  spent  by  laboratory  personnel  in  computer  operation 
does  not  include  functions  performed  at  remote  terminals 
or  input  devices. 
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Hypothesis  30;  Kinds  and  numbers  of  tests  sent  out  to  other 
laboratories  will  change,  for  an  equal  number 
and  distribution  of  test  requests,  an  equal 
mix  of  outside  laboratories  used,  and  an 
equal  distribution  of  tests  among  them. 

The  data  required  for  this  hypothesis  are  routinely 
recorded  in  the  laboratory  and  will  be  taken  directly  from 
the  laboratory  records.  Procedures  may  be  tested  at  the 
test  sites  rather  than  during  the  pretest  week. 

Procedure 

1.  For  each  test  site,  MET— compile  a list  of  tests 
routinely  sent  out,  the  facilities  to  which  tests 
are  routinely  sent,  and  the  cost  for  each  test. 

2.  MET — obtain  from  laboratory  records  the  number  of 
tests  sent  to  each  facility  during  the  week  immedi- 
ately preceding  collection  of  the  data  (7  days,  not 
including  the  day  of  collection)  (Form  10, 

Attachment  A) . 

3.  MET/ANSER — review  experiences  in  obtaining  the 
required  information,  identify  problems,  and  revise 
methods  and  forms  used. 

Hypothesis  31;  Kinds  and  numbers  of  tests  sent  to  the 

laboratory  being  studied  by  other  labora- 
tories will  change,  for  an  equal  number  and 
distribution  of  test  requests  at  the  labora- 
tories sending  the  test  requests. 

Comments,  procedures,  and  forms  are  those  identified 
for  Hypothesis  30  changing  "tests  sent  out"  to  "tests  received. 

Hypothesis  32;  The  number  of  stat  requests  will  change,  for 
an  equal  number  and  distribution  of  test 
requests. 

The  data  collection  for  this  hypothesis  is  counting 
Btat  requests.  It  will  be  done  during  the  pretest  week. 


Procedure 


MET— obtain  number  of  stat  requests  per  day  by 
laboratory  section  by  collecting  it  from  each  section  at 
the  end  of  each  day  (Form  6,  Attachment  A). 

Hypothesis  33;  Time  that  laboratory  staff  members  work  in 
addition  to  normal  duty  hours  will  change, 
for  an  equal  number  and  distribution  of  test 
requests. 

This  hypothesis  will  be  investigated  by  means  of  survey 
schedules  (see  Section  II,  this  Appendix  D) . 

Hypothesis  34;  Usability  of  laboratory  reports  to  physicians 
will  change. 

This  hypothesis  will  be  investigated  by  means  of  survey 
schedules  (see  Section  II,  this  Appendix  D) , 

Hypothesis  35;  Time  to  provide  patient  service  at  the  lab- 
oratory reception  desk  will  change,  for  an 
equal  number  and  distribution  of  test  requests. 

The  method  data  collection  for  this  hypothesis  will  be 
tested  during  the  pretest  week.  Patient  service  times  will 
be  observed  and  recorded  during  specified  time  intervals. 

Procedure 

★ 

1.  ANSER — randomly  select  intervals  for  observation. 

2.  MET — observe  patient  reception  area  during  specified 
intervals  and  record  service  times  (Form  11, 

Attachment  A) . 

3.  MET/ANSER — review  observation  experience,  identify 
problems,  and  revise  procedures  and  forms. 

Hypothesis  36;  Deleted 

* One  question  to  be  answered  in  the  pretest  is  whether  a 
single  observer  can  record  data  for  this  hypothesis  as 
well  as  for  Hypotheses  2,  3,  4,  and  5 during  the  same 
observation  interval. 
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Hypothesis  37;  Time  the  patient  spends  at  the  laboratory 

will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

Procedure 


1.  ANSER — specify  the  sample  of  patients  (e.g.,  all 
patients  who  arrive  during  random  time  intervals) . 

2.  MET — obtain  arrival  and  departure  time  for  the 
selected  sample  of  patients  (Form  12,  Attachment  A) „ 

A form  will  be  time  and  date  stamped  and  given  to 
the  patient  when  he  arrives.  It  will  be  collected 
and  stamped  again  as  he  departSo 

3.  MET/ANSER — review  collection  experience,  identify 
problems,  and  revise  procedures  and  forms. 

4.  Repeat  2,  3 loop  once  daily  or  after  each  observation 
as  needed  or  possible. 

Hypotheses  38,  39,  40,  41,  42,  and  43;  Acceptance  and 

Satisfaction 

38.  Patient  satisfaction  with  the  laboratory 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

39.  Physician  acceptance  of  the  laboratory  will 
change,  for  an  equal  number  and  distribution 
of  test  requests. 

40.  Laboratory  staff  satisfaction  will  change, 
for  an  equal  number  and  distribution  of 
test  requests. 

41.  Outpatient  clinic  staff  acceptance  will 
change,  for  an  equal  number  and  distribution 
of  test  requests. 

42.  Medical  records  department  staff  acceptance 
will  change,  for  an  equal  number  and  distri- 
bution of  test  requests. 

43.  Inpatient  nursing  unit  staff  acceptance  will 
change,  for  an  equal  number  and  distribution 
of  test  requests. 
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These  hypotheses  concern  the  way  various  groups  of 
persons  who  are  affected  by  the  laboratory  perceive  its 
operations.  The  hypotheses  will  be  investigated  by  means  of 
survey  schedules  (see  Section  II,  this  Appendix  D) . 

Hypothesis  44:  The  number  of  transcription  errors  will  change, 

for  an  equal  number  and  distribution  of  test 
requests . 

This  hypothesis  is  to  be  investigated  in  part  through 
survey  schedules  (see  Section  II,  this  Appendix  D) . In 
addition,  application  of  standard  error  rates  based  on 
"opportunities  for  error"  is  being  investigated.  Actual 
data  on  error  opportunities  will  be  collected  as  part  of 
the  data  specified  for  Hypothesis  9.  Therefore,  no  pre- 
test data  collection  procedure  is  given  for  this  hypothesis. 

Hypothesis  45;  Reliability  of  the  medical  record  will  change, 
for  an  equal  number  and  distribution  of  test 
requests. 

The  data  to  be  collected  for  this  hypothesis  are  numbers 
of  laboratory  reports  that  have  been  filed  in  outpatient  and 
inpatient  medical  records  3 days,  5 days,  and  7 days  after 
they  are  completed.  Th®se  numbers  are  to  be  determined  for 
samples  of  laboratory  reports.  Methods  for  collecting  this 
data  will  be  tested  during  the  pretest  week. 

Procedure 

1.  ANSER — specify  methodology  for  selecting  random 
samples  of  completed  result/request  slips  in  the 
laboratory. 

2.  Laboratory  personnel — stamp  time  and  date  on  completed 
request/result  slips  when  they  are  brought  to  the 
distribution  point  in  the  laboratory  (e.g.,  the 
reception  desk)  . 
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3.  Outpatient  Medical  Records  staff — stamp  date  on 
request/result  slips  when  they  are  filed  in  the 
record. 

4.  MET — obtain  the  specified  samples  of  completed 
reports.  Make  photocopies  of  the  selected  request/ 
result  slips  and  assign  each  slip  a number  (Form  7, 
Attachment  A) . (Testing  this  procedure  is  also 
included  in  the  pretest  procedure  specified  for 
Hypothesis  46.) 

5.  MET — hold  the  copied  slips  until  at  least  7 days 
after  their  completion  date,  then  obtain  the 
following  information  from  the  outpatient  medical 
records  of  the  patients  whose  slips  are  in  the 
Scimple : 

(a)  Is  the  record  in  the  record  room? 

(b)  If  yes  to  (a) , is  a copy  of  the  slip  filed 
in  the  record? 

(c)  If  yes  to  (b) , how  many  days  after  completion 
by  the  laboratory  was  the  slip  filed?  (Form  14, 
Attachment  A) . 

6o  MET/ANSER — review  experiences  in  obtaining  the 
required  samples  and  in  surveying  the  outpatient 
medical  records;  identify  problems  and  revise  the 
methods  and  forms  used. 

7.  Continue  4,  5,  6 loop  until  methods  and  forms  are 
finalized. 

Hypothesis  46;  Turnaround  time  will  change,  for  an  equal 
number  and  distribution  of  test  requests. 

For  this  hypothesis,  data  will  be  collected  on  the  times 
of  occurrence  of  four  events  in  request  turnaround.  Two  of 


these  are  the  times  that  laboratory  teams  return  from 
inpatient  specimen  collection  rounds  and  the  times  that  result 
distribution  personnel  leave  the  laboratory  to  distribute 
completed  request/result  slips.  The  other  two,  for  individual 
request/result  slips,  are  time  of  arrival  in  the  laboratory 
and  the  time  the  result  slips  are  ready  for  distribution  from 
the  reception  area  or  other  distribution  point.  The  proce- 
dures for  collecting  this  data  will  be  tested  during  the 
pretest  weeK. 

Procedure 

1.  MET — record  daily  the  times  that  laboratory  specimen 
collection  teams  return  to  the  laboratory  (excluding 
time  of  return  for  special  stat  collection  trips) . 
Have  the  teams  sign  in  (Form  14  or  15,  Attachment  A). 

2.  MET — record  times  of  departure  of  laboratory  teams 
distributing  results  and  of  personnel  from  other 
parts  of  the  hospital  who  are  collecting  result 
slips.  Have  personnel  sign  out  (Form  14  or  15, 
Attachment  A) . 

3.  Laboratory  personnel — time  and  date  stamp  request/ 
result  slips  when  they  arrive  in  the  laboratory  and 
again  when  the  completed  slips  are  brought  to  the 
distribution  point  in  the  laboratory. 

4.  ANSER — specify  methodology  for  selecting  a random 
sample  of  completed,  time  and  data  stamped  slips. 

5.  MET — obtain  the  specified  Scimple  of  request/result 
slips.  Make  photocopies  of  the  selected  slips  and 
assign  each  slip  a number  (Forms  7 and  16,  Attach- 
ment A) . (Testing  this  procedure  is  included  in  the 
pretest  procedure  specified  for  Hypothesis  45.) 
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а.  MRT/ANSER — review  experiences  in  recording  the  times 
specified;  identify  problems  and  revise  methods  and 
forms  used. 

7.  Continue  1,  2,  3,  4,  5,  6 loop  until  methods  and 
forms  are  finalized. 

Hypothesis  47;  Quality  of  patient  care  will  change,  for  an 

equal  number  and  distribution  of  test  requests. 

Information  concerning  patient  care  will  be  obtained  by 
means  of  survey  schedules  (see  Section  II,  this  Appendix  D) . 

Hypothesis  4R;  Numbers  and  kinds  of  reports  produced  will 
change . 

Most  of  the  data  needed  for  this  hypothesis  is  included 
in  the  data  to  be  collected  for  the  preceding  hypotheses. 

During  the  pretest  week,  any  reports  not  covered  in  the 
preceding  hypotheses  will  be  identified. 

1.  MET — identify  and  determine  frequency  of  any  reports 
produced  that  are  not  included  in  the  preceding 
hypotheses . 

2.  ANSER — specify  methods  for  counting/sampling  addi- 
tional reports  identified  as  appropriate. 

3.  MET — apply  the  counting/sampling  methods  specified 
in  2 , if  they  have  not  been  tested  in  the  pretest 
procedures  for  preceding  hypotheses. 

4.  MET/ANSER — review  experiences  in  using  the  methods 
of  counting/sampling,  identify  problems,  and  revise 
the  methods  as  needed. 

5.  Continue  3,  4 loop  until  methods  are  finalized, 

б.  MET^-compile  a list  of  all  reports  and  the  number 
produced  per  day,  week,  month,  etc. 
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Hypothesis  49;  Numbers  and  kinds  of  physician  requests  for 
outpatient  tests  will  change,  for  an  equal 
number  of  outpatient  visits,  for  the  same 
distribution  of  utilization  by  outpatient 
departments . 

The  data  required  for  investigating  this  hypothesis  are 
laboratory  workload  for  outpatients  and  workloads  of  the  out- 
patient clinics.  Methods  of  collecting  these  data  will  be 
tested  during  the  pretest  week. 

Procedure 

1.  MET — record  number  of  tests  requested  by  day, 
department,  and  kind  of  test  for  outpatients 
(Form  6,  Attachment  A). 

2.  MET — record  number  of  visits  per  day  for  each  out- 
patient clinic.  Obtain  2 or  3 times  during  the 
pretest  week  from  clinics  (Form  17,  Attachment  A). 

3.  MET/ANSER — review  experiences  in  obtaining  the 
required  information,  identify  problems  and  revise 
the  forms  and  methods. 

4.  Continue  1,  2,  3 loop  until  forms  and  methods  are 
finalized. 

Hypothesis  50;  Numbers  and  kinds  of  physicians'  requests 

for  inpatient  tests  will  change,  for  an  equal 
number  of  total  hospital  inpatient  days,  for 
the  same  distribution  of  patients  by  in- 
patient category. 

The  data  required  for  investigating  this  hypothesis  are 
laboratory  workload  for  inpatients  and  the  hospital's  inpatient 
workload.  Methods  of  collecting  these  data  will  be  tested 
during  the  pretest  week. 

Procedure 

1.  MET — record  number  of  tests  requested  for  inpatients 
by  day,  department,  and  kind  of  test  (Form  6, 
Attachment  A) . 
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2.  MET — record  daily  inpatient  census  by  inpatient 

★ 

category,  and  by  ward,  and  the  number  of  admissions 
and  of  discharges  per  day.  Obtain  2 or  3 times 
during  the  pretest  week  from  the  registrar's  office 
(Form  18,  Attachment  A). 

3.  MET/ANSER — review  experiences  in  obtaining  the 
required  information,  identify  problems  and  revise 
the  forms  and  methods. 

4.  Continue  1,  2,  3 loop  until  forms  and  methods  are 
finalized. 

Hypothesis  51;  Time  required  to  complete  laboratory  test 
request  slips  outside  the  laboratory  will 
change,  for  an  equal  number  and  distribution 
of  test  requests. 

The  method  to  be  used  in  measuring  this  time  is  activity 
charting  and  application  of  time  standards.  The  charting 
specified  in  the  pretest  procedure  will  not  necessarily  be 
done  during  the  pretest  week.  Collection  of  data  for  apply- 
ing the  standards  (number  of  request  forms  by  inpatient  or 
outpatient)  will  be  tested  during  the  pretest  week. 

Procedure 

1.  MET — chart  the  task  of  completing  request  forms. 

The  procedure  for  chart  development  is  as  described 
for  preceding  hypotheses  (see  for  example  Hypothesis 
6)  . 

2.  The  method  of  recording  the  number  of  request  slips 
by  day  for  inpatients  and  outpatients  has  been 
specified  for  preceding  hypotheses.  (See  procedure 
for  Hypothesis  6,  Form  6,  Attachment  A). 

* Same  categories  used  by  Air  Force  hospitals, 
t See  note  on  page  224. 
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. MET — obtain  supervisors'  estimates  of  fraction  of 
task  time  contributed  by  each  category  of  personnel 
(Form  3,  Attachment  A). 

Hypothesis  52;  The  numbers  of  improperly  completed  test 

request  slips  arriving  at  the  reception  desk 
will  change  for  an  equal  number  and  distri- 
bution of  test  requests. 

To  investigate  this  hypothesis,  a sample  of  size  n (n  to 
be  determined)  of  arriving  request  slips  will  be  checked  for 
errors.  This  data  collection  will  be  tested  during  the  pre- 
test week. 

Procedure 

1.  ANSER — specify  a number  of  slips  to  be  included  in 
the  sample  and  method  of  selecting  the  n slips  for 
the  sample  (For  example,  all  arriving  slips  until 
n are  observed  or  all  arriving  slips  until  n/p  are 
observed  for  p observation  periods) . 

2.  MET — record  all  errors  for  the  n slips  (Form  19, 
Attachment  A) . 

Hypothesis  53;  Time  required  by  supervisors  (laboratory 
officers)  in  responding  to  inquiries, 
complaints,  errors  detected  before  test 
results  are  reported,  and  errors  in  test 
results  reported  to  physicians  will  change, 
for  an  equal  number  and  distribution  of 
test  requests. 

Time  spent  in  these  tasks  will  be  estimated  and/or 
recorded  by  the  supervisors  involved.  This  data,  collection 
will  be  tested  during  the  pretest  week. 

Procedure 


1.  Supervisors  record  or  estimate  actual  time  spent 
by  day  in  responding  to  inquiries,  errors,  and 
complaints  (Form  2,  Attachment  A) . 
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2.  MET — obtain  estimates/records  from  supervisors  daily. 

3.  MET/ANSER — review  experiences  in  obtaining  the 
estimates/records,  revise  methods  and  forms  used. 

4.  Continue  1,  2,  3 loop  to  finalize  procedures  and 
forms . 

Hypotheses  54  through  59:  These  hypotheses  were  added  to 

the  study  design  after  the  pretest 
was  completed.  Hence,  they  were 
not  pretested. 
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ATTACHMENT  A 

FORMS  FOR  DATA  COLLECTION 
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List  of  Forms 


Form 

1 

Administrative  Report  Summary 

Form 

2 

Timesheet 

Form 

3 

Task  Time  Distribution  Form 

Form 

4 

Record  of  Telephone  Inquiries 

Form 

5 

Record  of  Inquiry  Response  Calls 

Form 

6 

Daily  Workload  Record 

Form 

7 

Identification  Slip  (for  samples 
laboratory  forms) 

of 

Form 

8 

Counting  Form 

Form 

9 

Counting  Form 

Form 

10 

Record  of  Tests  Sent  Out  or  Received 

Form 

11 

Reception  Area  Observation:  Patient  Service 

Time 

Form 

12 

Patient  Time  at  the  Laboratory 

Form 

13 

Outpatient  Medical  Records  Survey 

Form 

14 

Record  of  Return  from  Specimen  Collection 
Rounds  or  Departure  for  Result  Slip 
Distribution 

Form 

15 

Record  of  Return  from  Specimen  Collection 
Rounds  or  Departure  for  Result  Slip 
Distribution 

Form 

16 

Turnaround  Time 

Form 

17 

Report  of  Outpatient  Workload 

Form 

18 

Report  of  Inpatient  Workload 

Form 

19 

Record  of  Errors  in  Arriving  Lab 

Request  Slips 

The  original  19  data  collection  forms  used  in  the  pretest 
are  not  enclosed  because  they  are  similar  to  the  final  forms 
reproduced  in  Appendix  B,  Section  VI. 
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ATTACHMENT  B 

METHODS  TIME  MEASUREMENT  (MTM) 
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Methods  Time  Measurement  (MTM)  is  a system  for  estimating 
the  time  required  to  perform  routine  repetitive  tasks  by 
studying  the  motions  of  a person  and  then  using  standard  data 
to  estimate  the  time  required  for  the  operation.  Master 
Clerical  Data  (MCD)  as  published  by  Serge  Birn  et  al.  (Reference 
1)  is  a catalogue  of  standard  elements  used  by  clerical  workers 
to  perform  office  activities  and  the  associated  time  to  perform 
each  element.  The  data  provide  standard  values  that  are  con- 
sistent and  compatible  with  clerical  activities. 

The  use  of  MCD  is  most  easily  understood  by  considering 
the  following  example. 

Figure  1 is  a simplified  description  of  the  clerical  tasks 
involved  in  the  collection,  clinical  analysis,  and  reporting 
of  the  results  of  a blood  test.  The  time  required  for  each 
clerical  operation,  taken  from  MCD,  is  given  in  Table  1.  As 
computed  in  this  simplified  example  using  MCD,  the  total  time 
in  clerical  activities  per  specimen  tube  processed  that  will 
be  saved  by  introducing  AFCLAS  is  0.014  hour. 
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TABLE  1 

METHODS  TIME  MEASUREMENT  (MTM) 


Flow  Number 
From  Figure  1 

Description 

Work 

Units 

Frequency 

Total  Units* 
Per  Action 

1 

Place  and  remove  laboratory 
request  from  desk  file 

37 

1 

37 

2 

Write  name  on  tube^  (upper  case 
letters) 

24 

2 

48 

2 

Write  name  on  tube^  (lower  case 
letters) 

19 

10 

190 

2 

Vv’rite  name  on  tube^  (punctuation) 

15 

1 

15 

3 

Place  and  remove  tube  from 
storage  rack 

87 

1 

87 

4 

Seated  shift  (swing  90°  when 
removing  and  replacing  tube 
in  storage  rack) 

122 

1 

122 

5 

Write  name  in  log^  (upper  case 
letters) 

24 

2 

48 

5 

Write  name  in  log^  (lower  case 
letters) 

19 

10 

190 

5 

+ 

Write  name  in  log  (punctuation) 

15 

1 

15 

6 

Affix  label  to  tube  with  3-digit 
ID  number 

49 

1 

49 

7 

Affix  label  to  request  slip  with 
3-digit  ID  number 

49 

1 

49 

8 

Carry  laboratory  request  to 
table  for  completing  work- 
sheets (assume  a batch  size 
of  10  requests) 

30 

1/10 

3 

9 

Prepare  worksheets 

18 

3 

54 

10 

Place  and  remove  laboratory 
request  from  desk  file 

37 

1 

37 

11 

Transfer  label  from  specimen 

136 

1 

136 

tube  to  tube  containing 
serum  or  plasma 


• One  Unit  = 0.00001  hour 
= 0.0006  minute 
= 0.036  second. 

+ Names  are  assumed  to  total  12  letters  (last  name,  first  name 
with  comma  between) . 
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TABLE  1 — c:ontinued 
METHODS  TIME  MEASUREMENT  (MTM) 


Flow  Number 
From  Figure  1 

Description 

Wor)t 

Units 

Frequency 

Total  Units* 
Per  Action 

12 

Transfer  test  results  from 
wor)^sheet  to  laboratory 

25 

3 

75 

request  slip  (read  3 digits 
and  write  3 digits) 

13  Place  and  remove  wor)<sheets  from  37  1/10  4 

des)t  file  (assume  one  worV:sheet 
for  all  tests  is  used  for  every 
10  patients) 


14 

Supervisor  initials  (wor)csheets) 

22 

3 (1/10) 

7 

15 

Place  and  remove  laboratory  re- 
request slips  from  des)c  file 

37 

1 

37 

16 

Supervisor  initials 
(laboratory  request  slip) 

22 

3 

66 

17 

File  copy  of  laboratory 
request  slip  in  laboratory 
file 

55 

1 

55 

Total 

Units : 

1,324 

* One  Unit  = 0.00001  hour 
= 0.0006  minute 
= 0.036  second. 
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